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trhe  objective  of  this  study  was  to  develop  paper-and-pencll  predictors  of  Navy 
and  Marine  recruiter  perf ormance  and  evaluate  their  validity.  Accordingly, 
several  measures  of  personality,  vocational  interests,  and  background  were 
prepared  (or  selected)  and  administered  to  a geographically  representative 
sample  totaling  329  Navy  and  118  Marine  Corps  recruiters.  Scores  on  the 
predictor  battery's  items  and  scales  were  correlated  with  performance  scores 
developed  from  supervisory,  peer,  and  self  ratings  and  from  production  data  -n_ 
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,Xi.e.,  numbers  of.  recruits  enlisted).  Estimated  cross-validities  for  pre- 
dictor composites  were  significantly  different  from  zero  for  four  of  the  five 
performance  criteria  in  the  Navy  sample.  They  ranged  from  .17  to  .31.  Cor- 
responding validity  estimates  for  the  Marine  Corps  sample  ranged  from  .22 
to  .38,  (p  < .01  for  three  criteria,  jj  < .05  for  two  criteria). 

Recommendations  from  the  study  Included: 

1.  Examine  the  predictive  validity  of  the  predictor  composites  developed 
in  this  project^ 

2.  Assess  the  potential  fakabllity  of  the  predictor  composites  J"  £•— « — 

f \ / 

3.  ' Develop  additional  paper -and -pencil  measure8  of  constructs  that  this 
study  suggests  are  valid  Indicators  of  Navy  and  Marine  Corps  recruiter  success. 
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This  study  constitutes  the  first  phase  in  the  development  of  techniques 
for  identifying  those  individuals  with  the  greatest  likelihood  of  perform- 
ing effectively  as  Navy  and  Marine  Corps  recruiters.  It  includes  evaluations 
of  existing  selection  procedures  as  well  as  experimental  procedures. 

Work  was  conducted  in  support  of  Advanced  Development  Subproject 
ZFN01.06  (Advanced  Navy  Recruiting  System)  and  Exploratory  Development  Task 
Area  ZF55.521.001.101  (Marine  Corps  Personnel  Resources  Management),  under 
the  joint  sponsorship  of  Navy  Recruiting  Command  and  Headquarters,  Marine 
Corps  (MPI-20). 

The  results  of  the  study  will  be  used  in  a second  phase  that  will  evaluate 
the  predictive  validity  and  fakabllity  of  some  of  the  predictor  composites 
examined. 

Appreciation  is  expressed  to  LCDR  Hank  Levein  of  the  Navy  Recruiting 
Command  and  to  Major  William  Blaha,  Headquarters,  Marine  Corps,  for  their 
skillful  coordination  efforts  in  providing  the  samples  used  in  the  study. 
Additionally,  special  thanks  go  to  the  participating  Navy  and  Marine  Corps 
recruiters  who  provided  their  time  and  expertise  to  aid  us  during  the  project. 

The  technical  monitor  was  Dr.  Norman  Abrahams. 
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SUMMARY 


Problem 

The  U.S.  Navy  and  Marine  Corps  are  very  concerned  about  the  number  and 
quality  of  young  persons  brought  Into  these  services.  Recruiters  charged 
with  the  responsibility  for  contacting  prospective  recruits  and  selling 
them  op  a Navy  or  Marine  Corps  career  are  clearly  the  key  personnel  for 
successful  recruitment  of  well-qualified  men  and  women.  Therefore,  the 
selection  of  recruiters  with  good  potential  for  success  on  this  job  becomes 
a vitally  important  problem. 

Objective  , 

The  objective  of  this  study  was  to  develop  pencll-and-paper  predictors 
of  Navy  and  Marine  Corps  recruiter  effectiveness  and  evaluate  the  validity 
of  these  measures. 

Approach  . 

Based  on  a literature  review  of  previous  military  recruiter  selection 
studies,  a trial  predictor  battery  of  personality,  interest,  and  biographical 
items  and  scales  was  developed.  The  battery  was  administered  to  a geographically 
representative  sample  totalling  329  Navy  and  118  Marine  Corps  recruiters.  Scores 
on  the  predictor  measures  were  correlated  with  performance  ratings  and  with 
an  objective  effectiveness  index  in  a concurrent  validation  design.  Rela- 
tionships between  the  various  predictors  and  performance  criteria  were 
assessed. 

Findings 

1.  Estimated  cross-'validities  for  predictor  composites  against  four  of 
the  five  performance  criteria  were  significantly  different  from  zero  at 
the  .01  level  for  the  Navy  sample.  These  estimated  validity  coefficients 
ranged  from  .17  to  .31. 

2.  Comparable  validity  estimates  for  the  Marine  Corps  sample  ranged 
from  .22  to  .38,  with  all  five  predictor  composite-performance  criterion 
relationships  significantly  different  from  zero  at  the  .05  level. 

Conclusions 

The  predictor  battery  developed  in  this  study  shows  promise  for  aiding 
Navy  and  Marine  Corps  decision-makers  in  selecting  recruiters  for  the  two 
services. 

Recommendations 

...  „ . ■ •' 

1.  Examine  the  predictive  validity  of  the  composites  developed  in  this 
research. 

2.  Examine  the  potential  fakability  of  the  selection  instruments, 

3.  Develop  additional  paper-and-pencil  measures  of  constructs  that  have 
proven  to  be  valid  predictors  of  recruiter  performance. 


vii 


CONTENTS 


Page 

INTRODUCTION 1 

Problem 1 

Purpose 1 

Background . . . 1 

Navy  Studies 1 

Marine  Corps  Studies  ......  5 

Amy  Studies 10 

The  Air  Force  Study 13 

Implications  of  These  Studies  for  the  Present  Project  16 

METHOD 17 

Summary  of  Methods 17 

Pretest  Procedures  17 

Pretest  Results 19 

Interrater  Agreement 19 

Leniency 22 

Restriction  of  Rsnge 22 

Halo 22 

Determining  Dimensionality  of  the  Ratings 22 

Developing  an  Objective  Production  Index  . 23 

Pretest  Performsnce  Criteria  23 

Using  Pretest  Dsta  to  Refine  Rating  Scale  Materials 23 

Using  Pretest  Responses  to  Revise  the  Trial  Predictor  Battery  . . 24 

Forming  Predictor-Criterion  Correlation  Hypotheses  Using 

Pretest  Data  26 

Assigning  Cross-Validation  Strategies  to  Different  Sections  of 

the  Predictor  Battery  27 

Personality  Scales 27 

SCII 27 

Biographical  Survey 27 

Description  of  the  Sample  for  the  Main  Study 28 

Procedures  for  the  Main  Study 28 

RK8ULT8 31 

Criterion  Development  Results  ........  31 

Interrater  Agreement  ..........  31 

Leniency  .........  31 

Restriction  of  Range  31 

Bale 31 

Determining  Dimensionality  of  the  Ratings 34 

Developing  the  Objective  Production  Index  34 

lx 


Page 

Performance  Criteria  37 

Belationahips  Between  Criteria  37 

Trial  Predictor  Battery  Validation  Results  . . 37 

Cross-Validation  Results  for  Preselected  Keys  . 38 

Monte  Carlo  Cross-Validation  Results  38 

Estimated  Validity  for  the  Final  Composites  Pooled  Together  . . 43 

Black-White  Comparisons:  An  Examination  of  Bias  and 

Differential  Validity  in  the  Navy  Sample 43 

CONCLUSIONS 47 

Performance  Measurement:  Differences  Between  the  Present  Study 

and  Previous  Efforts 47 

Relationships  Between  Predictor  Composites  and  Recruiter 

Performance:  Some  Perspective  .......  48 

Comparison  of  16PF  and  Predictor  Composite  Validity  Results  ...  50 

HECOMMENDATIONS 53 

Studying  the  Predictive  Validity  of  the  Predictor  Composite  ...  53 

Studying  the  "Fakablllty"  of  the  Predictor  Battery  53 

Developing  Additional  Measures  of  Valid  Constructs  54 

REFERENCES 55 

APPENDIX  A— 16PF  VALIDITY  STUDY A-0 

APPENDIX  B— SCALE  DEVELOPMENT  STRATEGIES  AND  DEFINITIONS  OF  RATING 

DIMENSIONS B-0 

APPENDIX  C— MAIN  SAMPLE  PERFORMANCE  RATING  SCALE  PACKAGE C-0 

APPENDIX  D— NAVY  RECRUITER  BIOGRAPHICAL  AND  OPINION  SURVEY  D-0 

APPENDIX  B— DESCRIPTION  OF  THE  MONTE  CARLO  CROSS-VALIDATION 

PROCEDURE B-0 

APPENDIX  F— HYPOTHESIZED  PREDICTOR-PERFORMANCE  CRITERION 

RELATIONSHIPS F-0 

APPENDIX  G-— IMPLEMENTATION  OF  THE  PREDICTOR  TEST  BATTERY  FOR  USE 

IN  SELECTING  NAVY  AND  MARINE  CORPS  RECRUITERS G-0 

DISTRIBUTION  LIST 


x 


LIST  OF  TABLES 

Page 

1.  Measures  Included  in  Pretest  Predictor  Battery  18 

2.  Pretest  Bating  Dimension:  Results  of  Analyses  20 

3.  Intercorrelations  and  Reliabilities  of  the  Final  Five 

Performance  Criteria — Pretest  Data 24 

4.  Correlations  Between  Pretest  Predictor  Battery  Personality 

Scales  and  the  Five  Pretest  Performance  Criteria  25 

5.  Interrater  Agreement  Indices  for  Self,  Peer,  and  Supervisor 

Ratings  ......  32 

6.  Means  and  Standard  Deviations  of  Ratings  Pooled  Across  the 

Self-Peer-Supervisor  Rating  Sources  33 

7.  Three-Factor  Varlmax  Rotated  Solution  for  Self -Peer-Supervisor 

Ratings  Pooled  Together:  Navy  and  Marine  Corps  Main  Sample 
Combined  35 

8.  Intercorrelations  and  Reliabilities  of  the  Five  Criteria  ...  36 

9.  Correlations  Between  Predictor  Battery  Personality  Scales 

and  the  Five  Performance  Criteria:  Main  Sample,  Navy  .....  39 

10.  Correlations  Between  Predictor  Battery  Personality  Scales 
and  the  Five  Performance  Criteria:  Main  Sample,  Marine 

Corps 40 

11.  Cross-Validation  Results  for  the  Personality  Composites  ....  41 

12.  Cross-Validation  Results  for  the  SC  1 1 Keys 41 

13.  Monte  Carlo  Results — SC II,  Navy  Sample 42 

14.  Monte  Carlo  Results — Biographical  Survey  42 

15.  Intercorrelations  of  Predictor  Composites/Keys  and 

Bstimated  Validity  for  the  Final  Composites  Pooled  Together  . . 44 

16.  Black-White  Differences:  Means  and  Standard  Deviations  on 

Predictor  Composites  and  Performance  Criteria  45 

17.  Black-White  Differences:  Validity  Coefficients  for  Final 

Predictor  Composites  46 

xi 


INTRODUCTION 


Problem 


The  U.S.  Navy  and  Marine  Corps  need  highly  qualified  young  people  to 
fill  a variety  of  jobs  In  the  two  services.  Skilled  recruiters  are  needed 
to  contact  qualified  prospects  and  sell  them  on  a Navy  or  Marine  Corps 
enlistment.  Although  training  and  competent  management  can  help  create 
an  effective  recruiter  force,  selecting  Individuals  with  excellent  potential 
for  success  as  recruiters  is  of  equal  Importance. 

Purpose 

The  purpose  of  this  study  was  to  develop  and  evaluate  a set  of  paper- 
and-pencil  measures  capable  of  predicting  the  performance  of  prospective 
recruiters.  These  measures  can  greatly  aid  the  Navy  and  Marine  Corps  re- 
cruiter selection  effort. 

Background 

The  selection  of  effective  recruiters  responsible  for  enlisting  quali- 
fied personnel  has  been  of  great  concern  to  all  branches  of  the  Armed 
Forces.  Selection  batteries  tapping  a wide  variety  of  Individual  differ- 
ences (e.g. , biographical  data,  occupational  interest,  personality,  and 
personal  preferences)  have  been  examined  to  determine  their  usefulness 
In  predicting  military  recruiter  success.  In  addition,  criteria  designed 
to  reflect  recruiter  effectiveness  (e.g.,  supervisor  ratings  and  production 
data)  have  been  employed  to  evaluate  tbe  validity  of  the  screening  batteries. 
This  section  presents  an  overview  of  recruiter  selection  studies  conducted 
by  the  Navy,  Marine  Corps,  Army,  and  Air  Force. 

Navy  Studies 

Wollack  and  Klpnls  (1960)  developed  a Navy  recruiter  selection 
battery  and  conducted  a concurrent  validity  study  to  assess  the  battery's 
usefulness  as  a screening  device.  The  predictor  battery  employed  by  the 
authors  was  composed  of  13  measures  designed  to  reflect  fluency  of  expres- 
sion, knowledge  of  the  Navy,  Interest  In  recruiting  activities,  and  general 
aptitude.  The  following  Instruments  constituted  the  predictor  battery: 

1.  Fluency  of  Expression:  Four  timed  Inventories  designed  to 
measure  a subject's  verbal  abilities.  (These  timed  Inventories  were 
administered  to  the  cross-validation  group  only.) 

a.  Thing  Listing  Test — requires  tbe  subjects  to  write  the 
names  of  as  many  things  as  possible. 

b.  First  Letters  Test—requires  the  sifcjects  to  write  as 
many  words  as  they  can  think  of  beginning  with  letter  ”b." 

c.  Four-Word  Combination  Test — requires  the  examinees  to 
compose  four-word  sentences  using  the  four  letters  presented  in  each  item 
as  the  initial  letters  of  the  words. 
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d.  Inventive  Opposite  Test — a word  is  presented  along  with  the 
first  letter  of  one  of  its  synonyms  and  the  first  letter  of  an  antonym.  The 
subject  is  required  to  fill  in  the  synonym  and  antonym  for  each  word  in  the 
series. 

2.  Knowledge  of  and  Bithuslasm  for  the  Navy 

a.  Navy  Knowledge — a 60-item  Inventory  designed  to  measure  a 
subject's  knowledge  of  Navy  history  and  traditions. 

b.  Career  Preference  Scale — 30  items  intended  to  measure  a sub- 
ject's attitude  toward  a Navy  career. 

c.  Career  Motivation  Survey — a 39-item  attitude  scale,  measuring 
attitudes  toward  various  aspects  of  Navy  life. 

3.  Vocational  and  Sports  Interests 

a.  Kuder  Preference  Record — a published  inventory  designed  to 
measure  a subject's  interest  in  10  broad  occupational  areas. 

b.  Sports  Inventory — a 50-item  inventory  dealing  with  rules  and 
plays  of  various  sports,  designed  to  measure  a subject's  Interest  in  athletic 
activities. 

4.  Aptitudes  in  four  areas  measured  by  the  Basic  Test  Battery. 

a.  General  Intelligence. 

b.  Arithmetic  Seasoning. 

c.  Mechanical  Comp rehens ion. 

d.  Clerical  Skills. 

The  validation  sample  consisted  of  410  active  recruiters  representing 
40  main  recruiting  stations,  substations,  and  branch  stations.  The  sample  was 
formed  by  contacting  the  commanding  officers  from  the  40  recruiting  stations 
and  asking  each  to  nominate  the  most  effective  (N  - 205)  and  least  effective 
(N  - 205)  recruiters  from  their  respective  stations.  The  nominations  were 
employed  as  the  criterion  measure  against  which  the  predictors  were  validated. 

To  validate  the  selection  battery,  it  was  administered  to  members  of 
the  sample  by  an  officer  from  each  recruiting  station.  The  completed  batteries 
were  returned  to  the  authors,  and  the  data  were  analyzed. 

The  Navy  Knowledge  Test,  Career  Preference  Scale,  Career  Motivation 
Survey,  and  Sports  Inventory  were  item-analyzed  using  the  effective  and  in- 
effective nominations  as  the  criterion.  Items  that  discriminated  between 
effective  and  ineffective  recruiters  beyond  the  .20  level  of  confidence 
were  retained  for  cross-validation.  Seven  items  from  the  Navy  Kaovledge 
Test  discriminated  beyond  the  .20  level,  as  did  15  items  from  the  Career 
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Preference  Scale,  ?4  items  from  the  Career  Motivation  Survey,  13  items 
from  the  Sports  Inventory,  and  four  scales  from  the  Kuder. 

Another  variable — Navy  service  racing  classification  (duty  prior  to 
serving  as  a recruiter) — was  analyzed  to  determine  whether  any  differences 
existed  between  the  effective  and  ineffective  groups.  Some  significant  dif- 
ferences were  found.  The  effective  recruiter  group  contained  larger  pro- 
portions of  men  in  Deck  (£  < .01)  and  Aviation  ratings  (j>  < .01).  The  in- 
effective group  contained  significantly  greater  proportions  of  men  in  Engine- 
ering and  Hull  ratings  (]>  < .01)  and  in  Construction  ratings  (j>  < .02). 

These  significant  rating  classification  differences  were  retained  for  sub- 
sequent cross-validation. 

The  sample  used  to  cross-validate  the  predictor  battery  consisted 
of  260  students  attending  a 6-week  recruiter  course.  The  predictor  bat- 
tery was  administered  to  the  cross-validation  group  while  they  were  attend- 
ing school.  Approximately  1 year  after  the  initial  testing,  when  all 
students  had  been  assigned  to  recruiting  duty,  ratings  were  collected  from 
the  two  supervisors  most  familiar  with  each  recruiter.  The  evaluation  forms 
contained  four  14-point  scales:  Technical  Competence,  Willingness  to  Work, 
Military  Manner,  and  Adaptability.  The  forms  also  required  the  supervisors 
to  answer  the  following  two  questions  using  5-point  scales: 

1.  If  you  had  been  given  the  chance,  would  you  have  recommended  the 
man  for  recruiting  duty? 

2.  Is  this  man  effective  in  recruiting  personnel? 

The  six  ratings  were  treated  as  six  different  criterion  measures. 
Recruiter  scores  on  each  dimension  were  computed  by  adding  the  two  supervisory 
ratings  together.  When  only  a single  supervisory  rating  was  available,  the 
scores  were  simply  doubled.  A total  of  222  recruiters  were  rated  by  two 
supervisors.  Interrater  reliabilities  ranged  from  .56  to  .66,  indicating  an 
acceptable  level  of  rater  agreement. 

Product-moment  correlations  between  the  predictors  selected  previously 
and  the  criterion  measures  were  computed  using  the  cross-validation  sample  of 
260.  Three  of  the  variables  significantly  predicted  the  overall  evaluation 
of  recruiting  effectivenes:  the  Kuder  Persuasive  Scale  (r  ■ .24,  £ < .01), 
the  Kuder  Scientific  Scale  (r  ■ -.17,  £ < .01),  and  the  Career  Motivation 
Survey  item  analysis  key  (r  - .13,  2.  < *05).  The  Kuder  Persuasive  Scale  also 
correlated  significantly  (£  < .01)  with  each  other  criterion  (except  Adapt- 
ability). In  addition,  the  Kuder  Scientific  Scale  correlated  significantly 
(£  < .01)  in  the  negative  direction  with  the  rating  dimensions.  Willingness 
to  Work  and  Recommendation  for  Recruiting  Duty,  and  the  Career  Motivation 
Survey  Key  correlated  significantly  (£  < .01)  with  Technical  Competence. 

None  of  the  other  predictors  correlated  significantly  with  any  of  the  criteria 
with  the  exception  of  one  fluency-of-expression  measure,  the  Pirst  Letters 
Test,  which  yielded  a correlation  of  -.18  (£  < .01)  with  Military  Manner.  A 
multiple  correlation  was  computed  using  the  Persuasive,  Scientific,  and  Career 
Motivation  Survey  predictors  against  the  criterion  measure  of  overall  recruiter 
effectiveness.  The  result  (R  - .26)  was  not  significantly  greater  than  the 
correlation  between  that  performance  dimension  and  the  Persuasive  Scale  alone. 


3 


Thus,  results  of  the  cross-validation  analysis  indicate  that  an  effective 
Navy  recruiter  has  persuasive  interests,  is  not  overly  interested  in  scholarly 
pursuits,  and  believes  in  the  value  of  a Navy  career. 

The  Wollack  and  Kipnis  (1960)  study  employed  a predictor  battery  con- 
taining a variety  of  tests  and  inventories.  Although  the  reliability 
of  the  criterion  nominations  was  reasonably  high,  only  a limited  number 
of  these  scales  and  item  keys  cross-validated  significantly.  This  may 
have  occurred  because  the  raters  were  making  their  evaluations  on  the 
basis  of  reputation  instead  of  performance  or  because  many  of  the  indi- 
vidual differences  that  are  truly  predictive  of  recruiter  success  simply 
were  not  included  in  the  battery. 

Krug  (1972)  administered  an  experimental  inventory,  the  16PF  personality 
Inventory,  to  Navy  enlisted  recruiters  and  officer  recruiters  to  determine  the 
usefulness  of  the  instrument  for  predicting  recruiter  effectiveness. 

The  Inventory,  which  was  administered  to  recruiters  in  41  recruiting 
stations  across  the  continental  United  States,  consisted  of  the  following  items: 

1.  The  16  PF  questionnaire.  Form  A,  1967  edition— a personality  test 
designed  to  measure  an  individual's  personality  in  terms  of  16  basic  factors. 

2.  A 25-item  experimental  supplement  including  7 items  designed  to 
measure  motivational  distortion  (a  lie  scale)  and  18  items  designed  to  measure 
strength  of  motivation  to  succeed  as  a recruiter. 

3.  Seven  biographical  items  yielding  information  about  years  of  service, 
age,  sex,  marital  status,  number  of  dependents,  years  of  formal  education,  and 
population  of  the  subject's  home  of  record. 

Commanding  officers  from  each  of  the  41  Navy  recruiting  stations  were 
asked  to  nominate  five  enlisted  recruiters  and  one  officer  recruiter  from  among 
the  top  50  percent  and  five  enlisted  recruiters  and  one  officer  recruiter  from 
among  the  bottom  50  percent  of  recruiters  presently  on  duty.  These  nominations 
of  recruiter  effectiveness  served  as  the  criterion  against  which  the  16PF,  sup- 
plemental, and  demographic  variables  were  validated. 

A stepwise  multiple  regression  of  the  several  predictor  variables  on 
the  criterion  for  the  enlisted  sample  resulted  in  a multiple  correlation  co- 
efficient of  .40,  significant  beyond  the  .01  level.  When  the  same  equation  was 
applied  to  the  officer  recruiter  cross-validation  sample  (N  - 74),  a multiple 
correlation  coefficient  of  .25  was  obtained  (g  < .05). 

According  to  these  results,  the  effective  Navy  recruiter  is  typically 
married,  has  more  years  of  formal  education,  tends  to  be  warm  and  outgoing, 
dominant,  aggressive,  self-assured,  and  relatively  conservative  politically. 

Although  the  results  from  the  cross-validation  sample  supported  the 
validity  of  this  instrument,  it  is  important  to  take  a closer  look  at  the 
regression  equation.  The  most  influential  variable.  Marital  Status,  was 
assigned  a high  positive  weight  in  the  prediction  equation.  Yet,  only  13  of 
the  383  enlisted  recruiters  in  the  sample  were  unmarried.  In  a selection 


situation,  a low  base-rate  of  unmarried  recruiters  would  make  that  variable 
less  useful  than  a variable  with  a more  balanced  base-rate. 

Abrahams,  Neumann,  and  Rimland  (1973)  employed  the  Strong  Vocational 
Interest  Blank  (SVIB)  to  develop  an  empirically  keyed  Recruiter  Interest  Scale 
(RIS)  for  use  in  selecting  Navy  recruiters.  The  first  step  in  deriving  the  key 
was  to  request  that  the  Commanding  Officer  in  each  of  the  42  main  recruiting 
districts  identify  the  five  least  effective  and  five  most  effective  recruiters. 
These  CO  nominations  were  used  as  the  criterion  to  develop  and  validate  RIS. 

Thus,  ten  SVIB  Inventories  were  mailed  to  each  of  the  42  districts  to  be 
administered  to  the  recruiters  selected  for  the  study. 

Complete  information  was  obtained  from  356  recruiters  representing  36 
of  the  recruiting  districts.  One-half  of  the  sample  (N  » 178)  was  used  to 
develop  the  RIS,  while  the  other  half  (N  * 178)  was  used  as  the  cross-validation 
group.  Each  of  the  399  SVIB  items  was  weighted  according  to  the  proportion  of 
most-  and  least-effective  recruiters  endorsing  the  response.  If  a response 
was  endorsed  more  often  by  the  most  effective  recruiters  it  was  weighted  +1. 
Conversely,  if  a response  was  endorsed  more  often  by  the  least  effective 
recruiters  it  was  weighted  -1.  The  115  items  with  the  largest  endorsement  dif- 
ferences were  included  in  the  SVIB-RIS-1. 

The  "holdout"  group  was  used  to  cross-validate  the  SVIB-RIS-1.  Re- 
sponses to  the  SVIB  items  were  assigned  weights  according  to  the  differential 
endorsement  patterns  of  the  previous  group.  The  scores  for  each  recruiter  were 
tabulated,  arranged  from  high  to  low  (high  scores  indicated  potentially  ef- 
fective recruiter  performance  and  low  scores  indicated  potentially  ineffective 
performance)  and  the  cross-validation  sample  was  then  divided  into  quartlles 
according  to  RIS  scores.  The  top  quart lie  (highest  RIS  scores)  contained  three 
times  as  many  effective  recruiters  as  did  the  bottom  quartile;  conversely,  the 
bottom  quartile  contained  three  times  as  many  Ineffective  recruiters  as  did  the 
top  RIS  group.  Clearly,  the  cross-validity  obtained  for  the  RIS-1  was  Impressive. 
Accordingly,  the  authors  recommended  using  the  SVIB-RIS-1  as  a selection  device 
to  exclude  low-scoring  individuals  from  recruiter  service.  Abrahams  et  al.  sug- 
gested that  a cutoff  score  would  largely  depend  on  the  number  of  applicants 
and  the  number  of  recruiters  needed.  The  authors  also  recommended  using  sup- 
plemental inventories  (e.g.,  biographical  items  found  effective  in  predicting 
sales  performance)  to  increase  performance  predictability. 

In  addition,  they  recommended  conducting  additional  research  with  the 
SVIB  to  provide  greater  stability  in  establishing  the  scoring  weights  and  in 
assessing  validity.  Abrahams  et  al.  stress  that  "a  better  criterion  of  recruiter 
effectiveness  must  be  developed"  and  that  additional  factors  of  recruiter  per- 
formance should  be  taken  into  account  in  future  validity  studies. 

Marine  Corps  Studies 

Bennett  and  Habers  (1973)  assessed  the  degree  to  which  individual 
recruiter  characteristics,  geographic  assignment,  and  "evaluation"  variables 
Influence  Marine  Corps  recruiter  productivity.  The  authors  collected  data  from 
259  Marine  Corps  recruiters  representing  29  recruiting  stations  throughout  the 
United  States.  The  information  was  collected  from  performance  records  at  the 
Marine  Corps  Recruiter  Depot,  the  enlisted  Master  File,  and  a survey  of 
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recruiters  and  their  jobs  conducted  by  the  Office  of  Manpower  Utilization. 

The  variables  were  divided  into  three  categories: 

1.  Selection  variables — General  Classification  Test  scores,  age,  race, 
level  of  education,  number  of  dependents,  previous  service  as  a career  planner 
or  drill  instructor,  method  of  assignment  to  recruiting  duty  (volunteer  or 
assigned),  and  opinion  about  recruiting  duty  as  a financial  hardship. 

2.  Deployment  variables — whether  the  recruiters  are  assigned  in  their 
home  states;  distance  from  home  state;  type  of  area  assigned  to — urban,  suburban, 
or  rural;  number  of  times  assigned;  hours  per  week  spent  on  recruiting;  and  per- 
centage of  time  spent  out  of  the  office  recruiting. 

3.  Evaluation  variables — number  of  months  the  individual  has  been  on 
his  current  tour  of  duty  and  the  recruiter's  percentile  rank  in  Marine  Corps 
Recruiter  Class. 

Gross  productivity  (l.e. , the  average  number  of  recruits  enlisted  per 
month)  was  employed  as  the  measure  of  performance  for  each  recruiter.  However, 
Bennett  and  Habers  noted  that  differences  in  gross  productivity  cannot  be 
completely  attributed  to  differences  in  the  performance  of  individual  recruiters, 
but  may  be  due  in  part  to  a variety  of  regional  differences  (e.g.,  the  number 
of  qualified  prospects  and  local  attitudes  toward  military  service).  For  this 
reason,  the  sample  was  divided  into  two  groups  on  the  basis  of  the  average 
regional  enlistment  rates  for  1971.  The  first  group  contained  those  recruiters 
from  high  enlistment  recruiting  stations  (N  - 109)  and  the  second  group  con- 
tained recruiters  serving  in  the  low  enlistment  areas  (N  ■ 150). 

The  authors  then  used  multiple  regression  to  determine  the  magnitude  of 
the  relationship  between  each  variable  and  recruiter  productivity.  A separate 
regression  equation  was  developed  for  each  enlistment  area  group.  In  the  high 
enlistment  area  group,  three  variables  were  found  to  be  significantly  related 
to  productivity.  The  first  variable,  type  of  area,  showed  that  urban  and  sub- 
urban recruiters  enlisted  more  persons  per  month  (£  < .01)  than  rural  recruiters. 
The  second  variable,  geographical  assignment,  showed  that  recruiters  stationed 
in  their  home  state  enlisted  more  persons  per  month  (£  < .01)  than  those  stationed 
more  than  500  miles  outside  their  home  state.  The  third  indicated  that  pro- 
ductivity was  negatively  related  to  Increased  reassignments  (j>  < .05).  These 
three  variables  accounted  for  35  percent  of  the  productivity  variance. 

In  the  low  enlistment  areas,  two  variables  correlated  significantly 
with  productivity.  The  hardship  variable  indicated  that  those  recruiters  who 
feel  recruiting  duty  is  a financial  hardship  enlist  more  persons  per  month 
(£  < .05)  than  those  who  do  not.  The  second  significant  variable,  service 
previously  as  a career  planner,  indicates  that  recruiters  with  prior  service 
enlisted  a greater  number  of  persons  per  month  (g  < .05)  tuan  did  those  who 
had  no  experience  as  career  planners.  The  two  variables  account  for  only  12 
percent  of  the  productivity  variance,  however. 

It  should  be  noted  that  the  regression  equations  obtained  in  the  study 
were  not  cross-validated,  and  thus  the  relationships  found  here  may  well  over- 
estimate the  true  usefulness  of  these  variables. 
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Larriva  (1975)  applied  the  Navy's  recruiter  selection  battery,  the 
16PF-M,  to  a Marine  Corps  sample  in  a concurrent  validity  Btudy. 

The  inventory,  which  was  administered  to  all  active  recruiters  from  the 
9th  Marine  Corps  District  (N  - 470),  consisted  of  the  same  items  used  in  the 
Krug  (1972)  study — the  16PF  questionnaire,  a 25-item  supplement,  and  7 bio- 
graphical items. 

The  16PF-M  inventories  were  mailed  to  the  9th  Marine  Corps  Recruiting 
command,  which  arranged  for  their  administration  to  the  recruiters.  The 
Recruiting  Command  also  made  arrangements  to  record  the  number  of  Non-Prior 
Service  (NPS)  accessions  (i.e.,  recruits  enlisted)  for  each  recruiter  through- 
out the  year  1974.  The  accessions  data,  which  were  employed  as  the  criterion 
measure,  were  provided  to  the  author  quarterly. 

After  all  the  completed  16PF-M  Inventories  were  returned  and  the  first- 
quarter  accession  data  were  reported,  Larriva  analyzed  the  data  using  the 
formula  developed  by  the  Navy  to  predict  recruiter  success.  This  analysis  in- 
dicated that  the  Navy  formula  was  not  a valid  equation  for  predicting  Marine 
Corps  recruiter  production. 

Larriva  conjectured  that  the  accessions  criterion  might  not  provide 
relevant  performance  information;  thus,  he  generated  several  performance 
indices  on  the  basis  of  assumptions  about  the  recruiting  environment  and 
geographical  differences  in  production.  Using  first-  and  second-quarter  ac- 
cession data,  each  performance  Index  was  employed  as  the  criterion  measure 
and  correlated  with  the  personality,  demographic,  and  experimental  variables 
of  the  16PF-M.  Larriva  then  examined  the  resulting  predictor-criterion 
relationships  and  selected  the  criterion  index  that  yielded  the  most  valid 
multiple  correlation  coefficient.  The  index  selected,  referred  to  as  Pig,  in- 
dicated the  number  of  accessions  of  urban  and  rural  recruiters  separately  and 
corrected  for  geographical  differences  in  relative  performance  of  recruiters. 
Accordingly,  the  validation  sample  was  split  into  an  Urban  Recruiter  Group 
(N  ■ 308)  and  a Rural  Recruiter  Group  (N  - 162)  and  a weight  correcting  for 
geographical  differences  was  applied  to  production  scores.  The  data  from  each 
group  were  analyzed  separately. 

Larriva  also  chose  to  generate  samples  of  urban  and  rural  recruiters 
that  contained  only  the  best  and  poorest  recruiters.  The  reduced  samples 
were  referred  to  as  the  Urban  Hi-Low  (N  - 122)  and  the  Rural  Hi-Low  (N  * 96) 
groups. 


The  Pig  criterion  waa  then  used  as  the  performance  measure,  and  step- 
wise multiple  linear  regression  was  utilized  to  generate  four  formulas  (one 
for  each  of  the  samples — Urban,  Urban  Hi-Low,  Rural,  and  Rural  Hi-Low)  for  pre- 
dicting recruiter  success  from  the  16PF-M  data. 

The  regression  technique  yielded  prediction  equations  and  multiple 
correlation  coefficients  for  each  sample — Urban  R - .43  (p  < .01),  Urban 
Hi-Low  R ■ .59  (p  < .01),  Rural  R - .33  (p  < .157,  and  Rural  Hi-Low  R - .44 
<£  < .42). 
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The  means  and  standard  deviations  of  the  predicted  and  actual  scores 
were  calculated  for  the  four  data  bases  and  were  used  to  establish  cutoff 
scores.  These  cutoff  scores  were  then  used  hypothetically  to  assess  the 
effectiveness  of  the  prediction  equations  to  select  or  delete  recruiters 
in  a cross-validation  sample. 

The  cross-validation  sample  consisted  of  98  recruiters  who  were  not 
included  in  the  validation  sample  but  who  had  a minimum  of  three  months 
accession  data.  As  prescribed  by  the  production  index,  the  saiqale  was 
split  into  an  Urban  Recruiter  Group  (N  - 66)  and  a Rural  Recruiter  Group 
(N  - 32),  and  effectiveness  scores  (the  Plfi)  were  generated  for  each  re- 
cruiter. 


The  urban  prediction  equations  were  applied  to  the  urban  cross- 
validation  sample.  The  results  of  the  prediction  equations  were  compared 
according  to  accuracy  and  expected  Increase  in  productivity.  The  same 
procedures  were  followed  to  compare  the  results  of  the  Rural  prediction 
equation  and  Rural  Hi-Low  prediction  equation  when  applied  to  the  rural 
cross-validation  sample.  Results  for  the  two  comparisons  are  listed  below: 

Urban  Urban  Hi-Low  Rural  Rural  Hi-Low 


Number  deleted  by  cutting 


score  24 

Number  below  mean  deleted  21 

Percent  correct  deletion  88 

Mean  production  score  of 

those  deleted  9 . 29 

Mean  production  score  of 

those  remaining  12.29 

Expected  percentage  in- 
crease in  productivity  10 


34 

12 

12 

26 

8 

10 

76 

75 

83 

9.91 

10.06 

9.02 

12.54 

12.39 

13.02 

12 

7.6 

13 

From  these  comparisons,  Larrlva  concluded  that  the  Urban  predictor 
equation  and  Rural  Hi-Low  equation  would  prove  successful  in  selecting 
Marine  Corps  recruiters.  The  author  also  suggests  thst  both  equations 
could  be  used  to  analyze  each  16PF-M  inventory,  thus  allowing  the  Marine 
Corps  Recruiting  Command  to  select  the  optimal  setting,  urban  or  rural, 
for  each  new  recruiter. 

In  summary,  the  Navy  prediction  equation  did  not  effectively  predict 
Marine  Corps  recruiter  success.  However,  when  recruiting  setting  (urban- 
rural)  and  geographical  differences  were  accounted  for  in  the  criterion, 
relatively  successful  prediction  equations  were  formed,  one  for  urban  re- 
cruiters and  one  for  rural  recruiters.  Cross-validity  results  suggest  that 
the  16PF-M  my  be  a useful  selection  device  for  screening  candidates  for 
the  Marine  Corpe  recruiter  job.  It  would  be  of  interest  to  apply  the  Navy 
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prediction  equation  for  the  16PF-M  to  the  cross-validation  sample  using 
the  Pig  criterion.  Apparently,  Larriva  never  correlated  the  Navy  predic- 
tion equation  scores  with  the  scores  on  the  Pig  criterion. 

The  selection  of  a criterion  in  this  research  is  of  interest. 

Larriva  essentially  selected  the  criterion  that  correlated  most  highly 
with  the  predictors.  A clearly  more  acceptable  (and  justifiable)  method 
is  to  define  the  criterion,  or  criteria,  as  precisely  as  possible  and  then 
to  select  a measure  that  provides  relevant  and  reliable  measurement  of  each 
criterion  without  regard  to  the  predictors.  The  cross-validation  procedure 
employed  in  Larriva’ s study  makes  less  severe  the  criticism  just  leveled, 
but  the  Pig  criterion  would  have  been  more  justified  as  a measure  of  effec- 
tiveness if  predictor  data  had  not  dictated  its  choice. 

Graf  and  Brower  (1976)  investigated  the  usefulness  of  the  Strong 
Vocational  Interest  Blank  for  selecting  Marine  Corps  recruiters.  SVIB 
Inventories  were  administered  to  98  recruiters  working  in  the  Los  Angeles 
and  Santa  Ana  areas.  Of  those  98,  77  recruiters  completed  and  returned 
the  SVIB  inventories.  Officers  in  the  Los  Angeles  and  Santa  Ana  recruiting 
stations  were  asked  to  provide  effectiveness  ratings  for  each  of  the  re- 
cruiters. These  ratings  consisted  of  a 3-point  scale  representing  below- 
average,  average,  and  above-average  recruiter  performance,  and  were  employed 
as  the  criterion  measure. 

The  authors  recognized  that  the  Navy  Recruiter  Interest  Scale  developed 
by  Abrahams  et  al.  (1973)  on  a sample  of  Navy  recruiters  might  predict  Marine 
Corps  recruiter  effectiveness  as  well.  Thus,  Marine  Corps  recruiter  SVIB 
responses  were  scored  using  the  RIS  scale  and  its  validity  was  evaluated. 

First,  a t-test  was  used  to  compare  the  rated  performance  of  recruiters 
from  the  upper  and  lower  35  percent  of  the  RIS  score  distribution.  The 
results  were  in  the  predicted  direction  (significant  beyond  the  .05  level); 
the  higher  the  RIS  score,  the  greater  the  likelihood  that  the  recruiter 
was  in  the  effective  group. 

Second,  recruiters  were  dichotomized  as  above  average  or  below  average 
according  to  the  supervisory  field  ratings,  and  the  two  groups'  RIS  scores 
were  compared.  A t-test  once  again  showed  a significant  difference  (beyond 
the  .05  level)  in  the  predicted  direction  (r^t  biserial  - .30,  ]>  < .05). 

Although  the  RIS  affectively  discriminated  between  above-average  and 
below-average  Marine  Corps  recruiters,  Graf  and  Brower  hypothesized  that  dif- 
ferences between  Navy  and  Marine  Corps  recruiters  might  warrant  a separate 
selection  device.  The  euthors  elected  to  develop  a Marine  Corps  Recruiter 
Interest  Scale  (MCRIS)  key  based  on  Marine  Corps  recruiter  responses. 

To  permit  development  of  a separate  key,  the  SVIB  was  administered  to  a 
second  sample  of  marine  Corps  recruiters  (N  ■ 91)  attending  the  Marine 
Corps  recruiting  school.  Two  years  after  the  SVIB  administration,  the 
students  were  located  to  collect  recruiter-effectiveness  ratings  from  their 
respective  supervisors. 

The  criterion  measure  of  recruiter  effectiveness  was  used  to  dichoto- 
mise the  combined  sample  (N  ■ 168,  consisting  of  77  recruiters  from  the 
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original  sample  and  91  Marine  Corps  students  at  the  recruiter  school)  into 
below-average  and  above-average  recruiters  based  on  the  supervisor  ratings. 
The  339  SVIB  items  were  then  weighted  according  to  the  differential  endorse- 
ment patterns  of  the  two  groups.  Items  were  included  in  the  MCRIS  if  en- 
dorsement differences  between  above-  and  below-average  recruiters  ex- 
ceeded 15  percent,  this  procedure  resulted  in  75  items  for  the  key.  (In- 
terestingly, only  13  of  the  75  items  in  MCRIS  are  also  found  in  the  RIS.) 

The  combined  Marine  Corps  recruiter  sample  was  scored  using  the  RIS 
and  the  MCRIS.  The  validity  coefficients  were  .67  for  the  MCRIS  and  .29 
for  the  RIS.  However,  these  two  coefficients  cannot  be  compared  directly 
because  the  RIS  validity  coefficient  reflects  the  key's  cross-validity, 
while  the  cross-validity  of  the  MCRIS  cannot  be  gauged  from  these  data. 

Graf  and  Brower  recomnend  assessing  the  MCRIS's  utility  as  a 
selection  device  by  evaluating  the  cross-validity  of  the  key  on  a larger, 
more  representative  sample  of  Marine  Corps  recruiters.  The  authors  also 
indicate  concern  for  their  criterion  measure  of  recruiter  effectiveness. 

They  recommend  attempting  to  develop  a more  reliable  method  of  measuring 
recruiter  performance. 

Army  Studies 

Brogden  and  Taylor  (1949)  reexamined  the  results  of  their  earlier 
study  designed  to  predict  Army  recruiter  effectiveness  and  evaluated  the 
validity  of  the  test  battery  using  a different  criterion  measure  of  recruiter 
effectiveness.  The  initial  study  had  employed  four  paper-and-pencll  predic- 
tors of  recruiter  performance,  including  a measure  of  vocational  Interests 
and  three  measures  tapping  interests,  hobbies,  and  backgrounds.  The  four 
predictor  scales  had  been  item-analyzed  against  a production  criterion — 
the  number  of  recruits  obtained  by  each  recruiter  per  hour  on  recruiting 
duty.  The  empirically  keyed  predictors  were  cross-validated  using  a sample 
of  475  recruiters,  yielding  a validity  coefficient  of  .18. 

Brogden  and  Taylor  postulated  that  the  low  validity  found  in  the 
Initial  study  was  due  to  the  unreliability  of  the  production  criterion  and 
to  the  contamination  potentially  present  in  such  global  objective  indices. 

The  authors  elected  to  use  a new  criterion:  a dichotomy  of  retention  vs. 
nonretention  on  recruiting  duty.  The  predictor  data  collected  in  the  initial 
study  was  used  to  evaluate  the  validity  of  that  information  for  predicting 
recruiter  turnover.  When  the  turnover  criterion  was  applied  to  the  summed 
scores  of  the  four  inventories,  it  yielded  a biserial  validity  coefficient 
of  .36. 


The  focus  of  the  Brogden  and  Taylor  study  was  on  the  criterion  used 
to  validate  predictors  used  for  selecting  Army  recruiters.  The  initial 
production  criterion  was  dismissed  because  it  was  thought  to  contain  many 
factors  not  associated  with  recruiter  effectiveness.  The  authors  recognised 
that  the  turnover  criterion  also  contained  some  extraneous  factors  unrelated 
to  recruiter  effectiveness  (e.g.,  completing  tour  of  duty,  requesting  a 
transfer).  When  the  new  criterion  was  correlated  with  the  predictors,  how- 
ever, a higher  validity  was  obtained.  This  increase  in  validity  may  be  due 
in  part  to  the  relatively  low  reliability  and  significant  contamination 
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associated  with  the  production  criterion.  It  is  also  possible,  of  course, 
that  measures  such  as  those  used  here  are  simply  better  predictors  of  turn- 
over than  they  are  of  performance. 

Brown,  Wood,  and  Harris  (1975)  sought  to  develop  a valid  criterion 
measure  of  recruiter  effectiveness  and  to  develop  an  improved  Army  recruiter 
selection  instrument.  In  the  first  phase  of  their  project,  the  authors 
focused  on  a criterion  measure  used  unsuccessfully  in  previous  recruiter 
studies — total  production  scores.  Brown  et  al.  noted  that  these  figures 
were  confounded  by  "opportunity  bias"  arising  from  territorial  factors 
that  fall  outside  the  control  of  the  individual  recruiter.  They  hypothe- 
sized that  if  territorial  factors  were  somehow  partlalled  out  of  the  pro- 
duction score  variance,  the  result  would  be  a more  realistic,  unbiased 
picture  of  recruiter  effectiveness. 

The  authors  began  by  identifying  territorial  variables  that  could 
bias  production  figures  (e.g. , the  average  number  of  enlistments  per  re- 
cruiter in  recruiter's  District  Recruiting  Command  or  DRC;  the  proportion 
of  territory  that  is  metropolitan,  suburban,  or  rural;  and  the  recruiters' 
experience).  A total  of  12  territorial  factors  were  identified  as  potential 
biasing  agents. 

The  next  step  was  to  obtain  a nationwide  random  sample  of  Army 
recruiters.  The  sample  was  acquired  by  requesting  each  commander  from  the 
five  Regional  Recruiting  Commands  (RRCs)  to  supply  names  of  100  recruiters 
from  their  respective  RRCs.  Commanders  selected  recruiters  randomly  accord- 
ing to  their  social  security  numbers.  In  addition  to  the  names,  commanders 
were  asked  to  supply  total  production  figures  for  each  recruiter  for  the 
period  from  July  to  December  1973.  The  territorial  information  of  each  DRC 
represented  by  recruiters  in  the  sample  was  compiled  by  Army  personnel  from 
available  records  and  from  a special  market  survey  they  conducted. 

The  12  factors  were  subjected  to  a stepwise  multiple  regression  with 
total  production  scores.  Three  variables  contributed  significantly  (£  < .05) 
to  the  prediction  of  production  scores — 'average  production  per  recruiter 
in  subject's  DRC;  proportion  of  all  enlistees  who  chose  the  Army;  and  the 
proportion  of  the  zone  that  is  suburban.  These  three  variables  accounted 
for  51  percent  of  the  production  score  variance. 

The  three  territorial  factors  were  used  to  develop  a multiple  regres- 
sion equation  that  yielded  predicted  production  scores  for  each  recruiter. 

The  predicted  scores  were  employed  to  compute  unbiased  production  scores. 

This  was  accomplished  by  computing  the  ratio  of  total  production  to  predicted 
production  scores  multiplied  by  100,  and  resulted  in  what  the  authors  termed 
Benchmark  Achievement  Scores  (BAS) . 

The  BAS  scores  thus  consisted  of  production  data  corrected  for  DRC 
production,  proportion  of  young  persons  joining  the  Army  in  the  DRC,  and 
the  proportion  of  the  DRC  that  is  suburban.  The  adjusted  scores  should  pro- 
vide relatively  unbiased  production  information,  an  index  of  productivity 
that  allows  for  differential  opportunity  to  recruit  successfully  because 
of  location. 
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The  BAS  scores  were  correlated  with  another  production  measure 
of  recruiter  effectiveness.  Single  Achievement  Scores.  These  scores  were 
derived  by  computing  a ratio  of  total  production  to  the  average  production 
in  the  DRC  (the  variable  that  accounted  for  most  of  the  variance  in  the 
multiple  regression  equation).  Brown  et  al.  reconmended  that  the  Simple 

Achievement  Scores  be  used  as  future  criterion  measures  because  they  corre-  i 

late  highly  with  the  BAS  (r  « .96)  and  are  easier  to  compute. 

In  the  second  phase  of  their  research.  Brown,  Wood,  and  Harris 
developed  an  Army  recruiter  selection  battery.  Inventories  were  selected 
for  the  battery  based  on  recruiter  job  behaviors  and  personal  characteris- 
tics. This  information  was  gained  from  interviews  with  79  recruiters  with 

high,  average,  and  low  records  of  success.  Personnel  from  Army  Recruiting  I 

Headquarters  were  also  consulted  about  traits  necessary  for  recruiter 

success. 

The  selection  battery  consisted  of  15  paper-and-pencll  inventories 
and  one  verbal  performance  test.  Below  is  a list  of  the  measures  included 
in  the  battery. 

1.  Verbal  Fluency — Recruiters  were  asked  to  simulate  a presenta- 

tion of  the  benefits  of  Army  life  to  a prospective  enlistee.  Each  presen- 
tation was  recorded  and  scored  by  computing  the  ratio  of  the  number  of  words  J 

spoken  during  the  first  2 minutes  of  the  presentation  to  the  number  of  "aha" 

spoken  during  the  same  length  of  time. 

2.  Sociability  Measures— Four  inventories  were  used  to  measure 
a recruiter's  sociability  and  aff illative  tendency. 

3.  Achievement  Motivation — Three  inventories  were  employed  to  tap 
the  tendency  to  work  hard  to  achieve  self-appointed  goals. 

4.  Empathy  Measures — Four  instruments  were  used  to  measure  the 
ability  to  understand  the  point  of  view  of  others  and  the  drive  to 

win  or  complete  a sale. 

5.  Rejection  Tolerance  Measure — One  paper-and-pencil  Inventory  was 
employed  to  measure  a recruiter's  tolerance  to  rejection,  rebuffs,  and  in- 
sults. 


6.  Responsibility  and  Maturity  Measures— Three  instruments  were 
used  to  tap  information  about  a recruiter's  ability  to  manage  his  personal, 
financial,  and  official  duties. 

When  it  became  necessary  to  select  subjects  for  the  battery  administra- 
tion, the  criterion  development  was  not  yet  completed.  Consequently,  Brown 
et  al.  devised  a Composite  Supervisory  Rating  procedure  to  select  two  extreme 
I groups  of  recruiters— the  highly  successful  and  the  very  unsuccessful.  Two 

DRCs  (the  highest  ranking  DRC  and  the  lowest  ranking  DRC,  based  on  objec- 
tives achieved)  were  selected  from  each  of  the  five  RRCa.  Five  supervisors 
from  each  DRC  were  asked  to  nominate  the  ten  best  and  ten  poorest  recruiters 
within  their  DRC.  The  five  recruiters  mentioned  most  often  as  the  beet  re- 
cruiters were  used  in  the  High  Criterion  Group,  while  the  five  recruiters 
mentioned  most  often  as  the  poorest  recruiters  were  placed  in  the  Low  Criterion 
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Group  (N  * 45  for  the  High  Criterion  Group,  and  N * 43  for  the  Low  Criterion 
Group).  The  4-  to  6-hour  battery  was  administered  to  the  High  and  Low 
Criterion  Groups  in  each  DRC,  and  additional  information  concerning  each 
recruiter's  race,  religion,  and  aptitude  scores  was  collected  from  Army 
personnel  files.  Means  were  computed  for  the  keyed  Inventories  and  the 
aptitude  scores  for  the  High  and  Low  Criterion  Groups  separately.  Hone  of 
the  personality  measures  or  aptitude  scores  differentiated  significantly 
between  the  two  groups;  however,  20  background  items  did  differentiate 
significantly  (j>  < .10)  between  them.  These  items  pertain  to  work  habits, 
styles  of  handling  finances  and  debts,  educational  background,  and  reaction 
to  challenging  or  stressful  situations.  The  verbal  performance  test  also 
differentiated  significantly  <£  < .10)  between  the  two  groups.  Brown  et  al. 
suggest  that  because  recruiters  are  a relatively  homogeneous  group  required 
to  meet  several  minimum  qualifications  (e.g.,  GCT  scores,  age,  and  rank) 
and  because  of  the  length  of  time  in  the  service  (in  this  study,  the  mean 
service  time  was  14  years),  the  recruiters  may  have  formed  similar  attitudes 
and  opinions,  thereby  limiting  the  variance  in  attitude,  personal  prefer- 
ence, and  personality  Inventory  scores.  Conceivably,  however,  the  low 
relationships  between  predictors  and  the  criterion  is  a result  of  the 
criterion  measure  used.  Utilizing  the  relatively  unbiased  production 
index,  the  Simple  Achievement  Score,  might  have  led  to  higher  validities. 

The  Air  Force  Study 


The  Air  Force  Recruiting  Service  recognised  the  need  for  a recruiter 
selection  device  and  in  1966  enlisted  the  service  of  Massey  and  Mullins  to 
develop  such  a selection  instrument.  The  authors  began  by  constructing  an 
eight-inventory  battery  designed  to  measure  qualities  such  as  empathy, 
surgency  (friendliness  and  sociability),  and  perseverance,  all  thought 
to  be  desirable  In  recruiters.  The  measures  are  listed  below. 

1.  Airman  Preference  Test — a 43-item  five-choice  test  designed 
to  measure  empathy.  The  five  choices  are  to  be  ranked  according  to  the 
average  basic  airman's  preferences. 

2.  CBI  Opinion  Questionnaire — 83  items  describing  a person's  be- 
havior, an  Individual's  opinions,  old  proverbs,  and  expressions  of  Interest. 
The  examinee  is  to  show  agreement,  disagreement,  or  no  opinion.  Three  scores 
(Complexity,  Ego  Strength,  and  Introversion)  were  considered  likely  predic- 
tors of  recruiter  success. 

3.  Community  Information — 43  items  to  gain  factual  information 
about  areas  of  former  residence.  This  was  designed  on  the  rationale  that 
a person  who  knows  more  about  the  community  is  one  who  is  more  interested 
in  it  and  consequently  one  who  should  be  a more  effective  recruiter. 

4.  Daacrlntiva  Adjective  Inventory-— 182  paired  adjectives  used  to 
describe  a person's  general  appearance,  usual  behavior,  and  relations  with 
others.  The  examinee  selects  the  one  of  the  pair  that  best  describee  his. 
Participants  were  scored  on  three  scales  (Surgency,  Cooperativeness,  sad 
Orderliness). 


5.  FCSRI-A — 190  Items,  pairs  of  statements,  that  refer  to  a 
person's  appearance,  usual  behavior,  and  relations  to  others.  Three 
scores  (Surgency,  Cooperativeness,  and  Orderliness,  the  same  three  vari- 
ables as  measured  by  the  Descriptive  Adjective  Inventory)  were  considered 
likely  predictors  of  recruiter  effectiveness. 

6.  Recruiter  Language — made  up  of  four  subtests: 

a.  15  five-choice  items — examinee  mist  select  the  initial 
letter  of  a missing  word.  The  definition  is  provided. 

b.  15  five-choice  items — examinees  must  select  the  correct 
arrangement  of  words  appearing  /in  progressive  order.  The  words  express 
varying  degrees  of  the  same  general  concept. 

c.  10  five-choice  items — examinee  oust  select  the  best 
sentence  with  respect  to  grammar  and  usage. 

d.  15  five-choice  items — a sentence  is  provided;  an  examinee 
must  select  the  interpretation  that  means  most  nearly  the  same  as  the 
sentence. 


7.  Texas  Social  Insight — 40-item  test  to  describe  ordinary, 
unusual,  or  embarrassing  situations.  The  examinees  are  to  select  the 
alternatives  they  see  as  the  correct  courses  of  action. 

8.  Word  Power — test  of  ability  to  call  to  mind  certain  words. 

▲t  a given  signal  the  examinee  records  as  many  words  as  possible  begin- 
ning and  ending  with  a given  letter  of  the  alphabet. 

In  a preliminary  validation  step,  the  battery  was  administered 
to  210  recruiter  students  prior  to  entry  in  the  recruiter  course.  Twenty- 
four  scores,  derived  from  the  eight  Inventories,  were  compared  to  the 
criterion  measure  of  grsduatlon  or  elimination  from  the  recruiter  course 
(178  graduated  and  32  were  eliminated).  Four  of  the  Bcores — Recruiter 
Language,  Community  Information,  FCSRI-A  (Surgency),  and  the  Texas  Social 
Insight  test — yielded  significant  point  blserlal  correlations  (£  < .01). 

A combination  of  the  Recruiter  Language,  Comunlty  Information, 
and  FCSRI-A  (Surgency)  scores  yielded  a multiple  correlation  of  .23 
(£  < .01).  (The  Texas  Social  Insight  Test  did  not  add  significantly 
to  the  multiple  correlation.)  The  three  inventories,  with  scoring 
weights  of  2,  1,  and  1 respectively,  were  employed  as  an  interim  selec- 
tion battery,  pending  further  validation  of  the  eight-inventory  composite. 

The  full  eight-inventory  battery  was  administered  to  s larger 
sample  of  recruiter  students  (N  « 1067)  prior  to  entry  in  the  recruiter 
course.  Background  information  Including  age,  education,  marital  status, 
□umber  of  dependents,  total  active  federal  military  service,  end  AQB 
scores  was  also  tapped.  Pass/fail  information  was  subsequently  ob- 
tained for  each  recruiter  student. 
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The  validation  sample  (N  - 965,  students  with  complete  Information) 
was  randomly  split  into  two  groups — sample  B ( N - 485),  the  computing 
sample;  and  sample  C (N  * 480),  the  cross-validation  group.  Twenty-four 
scores,  derived  from  the  eight  inventories,  were  generated  for  sample  B 
recruiters.  These  scores,  along  with  the  background  variables,  were  com- 
pared to  the  recruiter  course  pass/fail  criterion.  Various  combinations 
of  scores  and  background  variables  were  also  compared  to  the  criterion 
measure.  No  combination  of  predictors  significantly  (<  .05)  increased 
prediction  obtained  using  the  three  scores  from  the  interim  battery. 

The  scoring  weights  of  the  interim  battery  (RSST-63)  were  altered  a 
number  of  times  and  used  to  score  sample  B recruiters.  These  scores 
were  validated  against  the  pass/fail  criterion.  No  alteration  of  regres- 
sion weights  surpassed  the  prediction  capabilities  of  the  interim  battery 
with  the  original  weights  (R  ■ .34,  £ < .01). 

In  addition,  the  regression  weights  of  the  predictor  scoreB  of 
sample  B (the  same  scores  and  weights  used  in  the  Interim  battery)  developed 
using  the  pass/fall  criterion  were  applied  to  sample  C responses.  The 
scores  of  sample  C recruiters  were  calculated  and  validated  against  the 
pass/fail  criterion.  This  yielded  a point  biserial  correlation  of  .21 
(£  < .05). 

The  same  procedures  were  followed  in  validating  the  predictor 
scores  against  a second  criterion — supervisor  field  ratings.  These  ratings 
were  obtained  from  supervisors  1 year  after  each  recruiter  was  placed 
on  duty.  Results  showed  that  no  combination  of  variables  of  the  entire 
set  significantly  predicted  the  field  ratings  at  the  .05  level. 

The  authors  also  investigated  the  capability  of  the  primary  Air  Force 
Specialty  Code  (AFSC)  variable  (service  specialty  from  which  a recruiter 
was  drawn)  to  predict  recruiter  success.  Recruiter  training  personnel 
postulated  that  those  specialties  that  require  much  interpersonal  contact 
would  produce  better  recruiters  than  those  specialties  that  require  little 
contact.  The  primary  AFSC  variable  was  divided  into  four  service  specialty 
variables  reflecting  various  degrees  of  required  Interpersonal  contact. 
Additionally,  12  Interaction  variables  were  generated  by  multiplying  each 
of  the  four  primary  service  variables  by  aptitude,  surgency,  and  depend- 
ability scores,  thus  creating  12  hybrid  individual  differences- AFSC  vari- 
ables. These  16  variables  were  then  correlated  with  the  field-rating 
criterion  measure  for  a sample  of  859  recruiter  course  graduates.  Only 
one  of  the  16  correlated  as  highly  as  .07  (£  < .05).  The  authors  there- 
fore discarded  the  primary  AFSC  as  a predictor  of  recruiter  success. 

A second  variable  regarded  as  a possible  predictor  of  field  re- 
cruiter success  was  recruiter  school  advisor  ratings.  These  ratings, 
collected  while  the  recruiters  were  in  recruiter  school,  consisted  of  a 
single  7-point  scale  ranging  from  outstanding  to  unsatisfactory.  The 
advisor  ratings  were  related  to  field  ratings,  yielding  a correlation 
of  .19. 


The  results  of  this  study  indicate  that  a battery  consisting  of 
the  Recruiter  Language  Test,  Community  Information  Inventory,  and  the 
FCSRI-A  (Surgency)  can  be  used  to  predict  recruiter  school  success,  but 
that  neither  this  battery  nor  recruiter  school  advisor  ratings  can  be  used 
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to  predict  field  recruiter  success.  The  authors  attribute  the  failure 
to  predict  field  recruiter  effectiveness  to  the  supervisory  performance 
rating  criterion  measure.  They  suggest  that  this  kind  of  rating  is  in- 
herently contaminated  by  several  rater  errors  such  as  halo  and  leniency. 
Another  difficulty  may  be  that  the  battery  the  authors  developed  is  simply 
more  appropriate  for  predicting  recruiter  school  success  than  field  re- 
cruiter effectiveness.  At  any  rate,  Massey  and  Mullins  believe  that 
a more  reliable  and  valid  measure  of  recruiter  effectiveness  should  be 
developed  to  provide  a fair  assessment  of  the  utility  of  any  paper-and- 
pencil  measures  purporting  to  predict  recruiter  performance. 

Implications  of  These  Studies  for  the  Present  Project 

One  difficulty  that  pervaded  virtually  every  study  reviewed  is 
the  "criterion  problem":  How  can  recruiter  performance  be  measured  in  a 
reliable  and  valid  manner?  Many  of  the  disappointing  results  obtained 
In  these  studies  may  in  fact  be  linked  to  poor  criterion  measurement. 

Thus,  it  seems  critical  to  measure  recruiter  performance  very  carefully 
in  the  present  research. 

Results  of  the  studies  discussed  in  this  section  also  provide  clues 
about  predictors  that  might  be  used  successfully  as  indicators  of  future 
effectiveness  as  a recruiter.  First,  the  work  of  Abrahams,  Neumann,  and 
Rimland  (1973)  suggests  that  vocational  interests  may  successfully  predict 
military  recruiter  effectiveness.  Results  from  the  Wallack  and  Klpnls 
(1960)  and  Graf  and  Brower  (1976)  studies  also  indicate  that  interest  in- 
ventories show  promise  as  Indicators  of  recruiter  effectiveness. 

The  Krug  (1972),  Larriva  (1975),  and  Brown,  Wood , and  Harris 
(1975)  studies  employed  with  sane  success  background  or  biographical 
variables  to  predict  recruiter  performance.  Although  there  la  undoubtedly 
no  single  background  ideal  for  recruiter  success,  it  is  possible  that  per- 
sons with  certain  kinds  of  backgrounds  and  experiences  are  more  likely  than 
others  to  perform  effectively  as  military  recruiters. 

Personality  or  trait  measures  were  also  considered  useful  as  pre- 
dictors of  recruiter  performance.  The  Krug  (1972)  and  Larriva  (1975)  studies 
found  reasonably  high  relationships  between  personality  scales  and  recruiter 
effectiveness. 

On  the  negative  side,  abilities  appear  unrelated  to  recruiter  per- 
formance. Intelligence  and  other  kinds  of  ability  measures  generally  failed 
to  predict  recruiter  success. 

Based  in  part  on  this  literature  review  of  test  validation  studies 
related  to  the  military  recruiter  job,  several  personality.  Interest,  and 
biographical  items  and  scales  were  selected  for  the  trial  predictor  battery. 
Additional  predictors  were  selected  based  upon  past  experience  in  other 
selection  studies  and  "clinical  hunches"  as  to  whet  kind  of  person  makes 
a good  Navy  or  Marine  Corps  recruiter.  These  "hunches"  were  based  primarily 
on  knowledge  of  the  Navy  and  Marine  Corps  recruiter  performance  requirements 
gained  from  a previous  study  (Borman,  Hough,  & Dunne tt a,  1976).  The  predic- 
tors selected  and  their  application  in  this  study  are  described  in  the  next 
section. 
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METHOD 


Summary  of  Methods 

A concurrent  validity  research  design  was  employed  In  an  attempt  to 
Identify  paper-and-pencil  predictors  of  Navy  and  Marine  Corps  recruiter 
effectiveness.  Performance  criteria  for  Navy  and  Marine  Corps  recruiters 
were  derived  from  supervisor,  peer,  and  self  ratings  on  performance  scales 
designed  to  evaluate  military  recruiters  (Borman,  Hough,  & Dunnette,  1976), 
and  on  additional  scales  developed  in  this  study.  Also  employed  as  a cri- 
terion was  an  objective  measure  of  productivity  that  indexed  the  number  of 
persons  each  recruiter  brought  into  the  Navy  or  Marine  Corps  over  a 6-month 
period.  This  index  was  adjusted  for  geographical  area. 

Paper-and-pencil  predictors  that  appeared  to  provide  a valid  indication 
of  future  success  as  a Navy  or  Marine  Corps  recruiter  were  selected  after 
a review  of  earlier  validation  studies.  A list  of  the  predictors  selected 
appears  in  Table  1.  The  predictor  battery  was  administered  to  a nation- 
wide sample  totaling  329  Navy  and  118  Marine  Corps  recruiters.  Performance 
ratings  and  the  productivity  index  were  gathered  for  the  same  persons  and 
relationships  between  predictors  and  performance  indices  were  evaluated. 

Also,  16PF  (Cattell,  1974)  scores  were  available  for  191  of  the  Navy 
personnel  in  the  sample  who  had  completed  that  inventory  before  being 
selected  for  recruiting  duty  or  during  training  for  the  job.  The  validity 
of  the  inventory  was  evaluated  by  correlating  16PF  scores  with  the  various 
performance  criteria  as  described  in  Appendix  A. 

The  methods  related  to  the  development  and  validation  of  the  predictor 
battery  are  detailed  below. 

Pretest  Procedures 


Before  the  trial  predictor  battery  and  performance  ratings  were  used 
in  the  main  study,  both  sets  of  materials  were  pretested  in  two  Navy  re- 
cruiting districts,  NRDs  Detroit  and  St.  Louis,  with  a total  of  62  Navy 
recruiters. 

In  each  location,  the  trial  predictor  battery  was  administered  and  reac- 
tions to  the  various  inventories  were  elicited.  Also  administered  during 
these  sessions  was  a performance-rating  scale  package  containing  four  sets 
of  rating  dimensions.  The  four  sets  of  dimensions  represent  four  different 
approaches  to  developing  performance  rating  scales.  Trait,  behavior,  multi- 
dimensional scaling,  and  factor  analysis  dimensions  were  included.  The 
reasons  for  including  so  many  scales  in  the  pretest  was  to  provide  an  oppor- 
tunity to  select  strategies  and  individual  scales  that  showed  the  most  promise 
for  yielding  reliable  and  valid  performance  information.  The  scale  develop- 
ment strategies  and  definitions  for  each  dimension  are  presented  in  Appen- 
dix B. 
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Table  1 


Measures  Included  In  Pretest  Predictor  Battery 


1.  Personality  Research  Form  (PRF)  (Jackson,  1967) 

Exhibition 

Order 

2.  California  Psychological  Inventory  (CPI)  (Gough,  1957) 

Dominance 
Sociability 
Social  Presence 
Socialization 

Achievement  via  Conformance 
Good  Impression 

3.  Differential  Personality  Questionnaire  (DPQ)  (Tellegen,  1976) 

Social  Closeness 
Hard  Work 
Authoritarianism 
Impulsiveness 

4.  Sales  Effectiveness  Scale  (SES)  (Dunnette,  1976) 

5.  Self-Description  Inventory  (SDI)  (Ghiselli,  1954) 

Intelligence 
Supervisory  Qualities 
Initiative 
Self-Assurance 

Perceived  Occupational  Level 
Dec  la  Ion-Making  Ability 
Soclometrlc  Popularity 
Perceived  Maturity 

6.  Strong-Campbell  Interest  Inventory  (SCII)  (Campbell,  1974) 

7.  Biographical  and  Opinion  Survey 


In  both  NRDs  three  sets  of  ratings  were  gathered.  Recruiters  were 
asked  to  provide  both  self  and  peer  evaluations.  The  peer  ratings  were 
directed  toward  all  recruiters  stationed  in  the  same  office.  Supervisory 
•valuations  were  made  by  persons  selected  from  the  Zone  Supervisor,  Chief 
Recruiter,  Enlisted  Processing  Officer,  Executive  Officer,  and  Commanding 
Officer  positions.  Those  supervisory  personnel  who  were  available  and 
seemed  most  qualified  to  rate  recruiter  performance  provided  the  supervisory 
ratings.  Two  supervisor  ratings  were  obtained  for  each  recruiter  whenever 
possible. 

In  addition  to  self,  peer,  and  supervisory  ratings,  objective  perfor- 
mance information  was  gathered  in  the  form  of  the  number  of  individuals 
each  recruiter  had  enlisted  in  the  Navy  over  the  6-month  period  from  May 
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to  October  1976.  Such  performance  data  were  available  for  54  of  the  62 
Detroit  and  St.  Louis  Navy  recruiters.  This  information  was  prepared  for 
analysis  by  first  standardizing  the  production  indices  according  to  NRD. 

That  is,  standard  scores  were  developed  for  each  recruiter  for  each  month 
by  standardizing  each  month’s  production  data  within  NRD.  This  operation 
corrects  for  unequal  opportunity  to  bring  persons  into  the  Navy  because  of 
geographical  location  of  the  NRD,  a practice  that  appears  warranted  based 
upon  results  of  Arima's  (1977)  recent  work. 

Thus,  for  the  Detroit  and  St.  Louis  pretest  sample  of  62  Navy  recruiters, 
there  were  available  for  each  recruiter  performance  ratings  on  28  different 
rating  scales,  production  data  over  a 6-month  period  (N  - 54),  and  predictor 
Information  based  on  recruiter  responses  to  the  trial  predictor  battery. 

Pretest  Results 


Performance  dimension  ratings  were  analyzed  to  assess  interrater  agree- 
ment along  with  leniency,  restrict ion-of-range , and  halo  errors.  Additionally, 
a factor  analysis  was  conducted  to  summarize  the  performance  rating  informa- 
tion. The  results  of  these  analyses  are  presented  in  Table  2.  Finally, 
to  assess  the  stability  of  the  production  index  chosen  as  an  objective  measure 
of  recruiter  performance,  its  reliability  or  consistency  was  examined  across 
the  6-month  period  for  which  data  were  available.  The  following  paragraphs 
describe  the  methods  and  objectives  of  these  tasks. 

Interrater  Agreement 

Intraclass  correlations  were  computed  depicting  the  Interrater 
reliability  of  all  possible  combinations  of  self,  peer,  and  supervisory 
ratings.  Since  the  magnitude  of  the  intraclass  reliability  coefficients 
depends  to  some  extent  on  the  number  of  raters  per  ratee,  the  coefficients 
obtained  were  adjusted  to  the  two-rater  level  using  the  Spearman-Brown 
Prophecy  Formula.  This  adjustment  makes  the  reliabilities  comparable  and 
thus  provides  a means  of  assessing  the  relative  degree  of  reliability  as- 
sociated with  each  combination  of  rater  sources. 

In  general,  the  self-supervisor  ratings,  when  pooled  together,  are 
most  reliable,  taking  the  number  of  raters  per  ratee  into  account.  For 
deciding  which  ratings  to  use  as  criteria,  however,  the  most  appropriate 
index  is  the  unadjusted  intraclass  coefficient.  This  is  because  the  un- 
adjusted intraclass  coefficient  reflects  the  reliability  of  the  mean  ratings, 
and  these  mean  ratings  are  the  data  that  are  actually  used  as  the  criterion 
measures.  Using  this  index  of  reliability,  the  peer  and  supervisory  ratings 
and  the  ratings  pooled  across  all  three  sources  provide  maximum  reliability. 
Further,  factor  analysis  of  the  self-peer-supervieor  ratings  pooled  together, 
which  is  described  later  in  this  section,  showed  the  clearest  factor  struc- 
ture. Therefore,  self,  peer,  and  supervisory  evaluations  were  employed 
In  subsequent  analyses.  Reliabilities  of  the  ratings  from  these  three 
sources  combined  appear  in  Table  2. 
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Table  2 (Continued) 


Leniency 


As  Borman , Dunnette,  and  Johnson  (1974)  and  Schvab,  Heneman,  and 
DaCotlls  (1975)  among  others  have  pointed  out,  one  cannot  directly  assess 
leniency  unless  the  "true"  criterion  scores  are  available  against  which 
to  compare  the  level  of  ratings.  For  example,  very  high  ratings  may  reflect 
actual  high-level  performance  of  ratees  rather  than  inflated  depictions  of 
these  individuals'  performance  effectiveness.  Yet,  it  can  generally  be 
Inferred  that  mean  ratings  near  the  scales'  highest  point  indicate  the 
presence  of  leniency.  Table  2 contains  the  mean  ratings  for  each  of  the 
scales.  The  means  of  5.31  to  6.50  on  a 9-point  scale  suggest  that  the 
ratings  are  probably  reasonably  free  from  leniency  error. 

Restriction  of  Range 

The  standard  deviations  in  Table  2 provide  some  indication  of  the 
restriction-of-range  error.  Again,  because  the  true  range  of  performance 
on  each  scale  is  unknown,  the  exact  degree  of  restriction  of  range  can- 
not be  determined;  however,  the  standard  deviations,  which  range  from  .97 
to  1.47,  suggest  that  the  range  of  the  ratings  is  at  a reasonable  level. 

Halo 

Assessing  precisely  the  degree  of  halo  error  is  impossible  unless 
the  true  degree  of  correlation  between  dimensions  is  known.  Then  the  ob- 
served correlation  between  ratings  on  the  various  dimensions  can  be  com- 
pared to  the  true  correlations,  with  higher  observed  correlations  indicating 
halo  error.  Since  there  is  no  way  to  know  the  true  correlations  between 
dimensions,  error  cannot  be  precisely  evaluated;  however,  the  magnitude  of 
the  observed  correlations  (mean  dimension  intercorrelations  - .63)  suggests 
that  some  halo  error  is  present  in  these  ratings.  Though  actual  perfor- 
mance on  these  dimensions  is  probably  correlated  to  some  extent,  almost 
certainly  it  does  not  correlate  as  highly  as  these  results  suggest.  This 
means  that  raters  were  forming  general  judgments  about  individual  ratees 
and  evaluating  each  ratee  at  much  the  same  level  on  different  dimensions. 

Thus,  overall,  the  leniency  and  restriction  of  range  of  these 
ratings  are  within  acceptable  limits  and  the  reliability  is  reasonably 
high  when  ratings  are  pooled  across  the  three  sources—self , peer,  and 
supervisors.  However,  a pronounced  halo  error  appears  to  pervade  the 
ratings < 

Determining  Dimensionality  of  the  Ratings 

Ratings  from  the  peer-supervisor  and  self-peer-supervisor  sources 
were  pooled,  and  the  ratings  from  all  of  the  dimensions  except  Overall  Ef- 
fectiveness were  intercorrelated  and  factor-analyzed.  The  three-factor, 
varlmax-rotated  solution  from  the  self -peer-supervisor  sources  proved  to 
be  most  readily  interpretable.  The  factor  loadings  appear  in  Table  2. 

The  factors  were  named  as  follows: 

I.  Prospecting  and  Selling. 

II.  Planning,  Organizing,  and  Administrative  Skills. 

III.  Developing  Good  Navy-Community  Relations  and  Expending 
Extra  Effort  to  Aid  Prospects  and  Recruits. 
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Developing  an  Objective  Production  Index 


The  consistency  of  the  production  index  was  assessed  by  computing 
an  intraclass  correlation  coefficient  that  compares  variability  in  the  pro- 
duction index  within  recruiter  with  total  variability  in  production  indices 
across  recruiters  for  the  6-month  period.  This  analysis-of-variance  pro- 
cedure yielded  an  intraclass  correlation  of  .88,  suggesting  that  the  pro- 
duction figures  are  stable  over  time.  (The  .88  intraclass  correlation  can 
be  Interpreted  as  the  reliability  of  the  mean  production  index  scores 
averaged  across  6 months).  Of  course,  high  reliability  does  not  ensure 
the  relevance  of  this  index  as  a performance  criterion,  but  it  is  a com- 
pelling summary  measure  of  "bottom  line"  recruiter  performance. 

Pretest  Performance  Criteria 


The  production  index,  along  with  scores  on  the  three  factors  dis- 
cussed previously,  provided  four  performance  criteria  against  which  to 
assess  the  validity  of  the  predictor  Inventory  scales  and  items.  A fifth 
criterion — the  overall  performance  rating  dimension — was  selected  because 
of  its  high  reliability  (r  - .81)  and  its  conceptual  appropriateness  as  a 
summary  performance  indicator. 

Table  3 presents  the  intercorrelations  between  these  five  criteria 
and  the  reliability  of  each.  The  overall  performance  rating  correlates 
highly  with  the  objective  production  index,  but  neither  of  these  summary 
performance  measures  correlates  highly  with  effectiveness  in  individual 
facets  of  the  job  as  represented  by  scores  on  the  three  factors.  This 
result  makes  good  sense.  Two  overall  performance  measures  might  be  expected 
to  correlate  more  highly  with  each  other  than  with  scores  on  individual 
aspects  of  performance.  Further,  Table  3 indicates  that  the  reliabilities 
of  the  factor  scores  are  reasonably  high. 

Overall,  the  five  criteria  appear  to  reflect  recruiter  performance 
effectively.  The  three  uncorrelated  composite  dimensions  represent  three 
conceptually  meaningful  aspects  of  performance,  and  the  objective  produc- 
tion index  and  the  overall  performance  rating  provide  two  summary  effec- 
tiveness measures. 

Using  Pretest  Data  to  Refine  Rating  Scale  Materials 

To  develop  a final  set  of  performance  rating  scales  for  the  main  study, 
the  content  of  the  various  rating  scales,  the  factor  structure  of  the  three- 
factor  varlmax  rotated  solution  discussed  previously,  and  the  reliabilities 
obtained  for  the  ratings  on  each  scale  were  examined.  First,  because  of 
the  considerable  effort  extended  previously  to  develop  behavior  summary 
scales  for  Navy  recruiters  (Borman,  Hough,  & Dunnette,  1976),  the  eight 
behavioral  performance  scales  presented  in  Table  2 (rating  dimensions  9 
to  16)  were  selected  for  use.  These  scales  appeared  to  have  good  potential 
for  providing  performance  ratings  of  a high  quality.  The  rest  of  the 
scales  (1  to  8 and  17  to  27)  were  considered  as  possible  supplementary 
performance  dimensions. 
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Table  3 


Intercorrelations  and  Reliabilities  of  the  Final 
Five  Performance  Criteria — Pretest  Data 


Performance 

Intercorrelat ions 

Reliabilities 

Criteria 

1 

2 

3 4 5 

.69a 

1. 

Prospecting  and  Selling 

— 

.65 

2. 

Planning,  Organizing,  and 
Administrative  Skills 

.06 

— 

.59 

3. 

Developing  Good  Navy- 
Community  Relations  and 
Expending  Extra  Effort  to 
Aid  Prospects  or  Recruits 

.04 

.05 

.81 

4. 

Overall  Effectiveness  as 
a Recruiter 

.77 

.46 

.30  — 

.88 

5. 

Product ion 

.50 

.19 

.10  .70  — 

Reliabilities  for  the  first  three  criteria  index  the  stability  of  the 
factor  scores  associated  with  these  dimensions. 


The  following  criteria  were  used  to  select  Beales  from  the  supplementary 
group  of  19:  (1)  adequate  lnterrster  reliability;  (2)  high  loading  on  one 

of  the  three  factors;  and  (3)  contribution  to  the  conceptual  completeness 
of  the  total  list  of  dimensions.  Using  these  guidelines  for  including  extra 
scales,  the  following  dimensions  were  selected:  Initiative,  Judgment, 
Confidence,  Achievement  Orientation,  Organization,  Warmth,  Expending  Extra 
Effort  to  Aid  Applicants  or  Recruits,  and  Developing  Productive  Relationships 
in  the  Community. 

The  content  of  these  scales  appeared  to  supplement  the  content  of 
the  behavior  summary  scales,  and  each  extra  scale  loaded  relatively  highly 
on  one  of  the  three  factors  from  the  pretest  analysis,  thereby  raising 
the  probability  of  a meaningful  factor  analysis  solution  in  the  main 
study.  Thus,  eight  behavior  summary  scales  and  eight  supplementary  scales 
selected  from  pretest  data  were  selected  for  use  in  the  main  part  of  the 
study.  Additionally,  Overall  Effectiveness  as  a Recruiter  was  included 
to  provide  an  overall  performance  scale.  See  Appendix  C for  the  rating 
scale  package  used  with  the  main  Navy  and  Marine  Corps  samples. 

Using  Pretest  Responses  to  Revise  the  Trial  Predictor  Battery 

Pretest  responses  to  the  predictor  inventories  were  used  to  refine  the 
trial  predictor  battery.  First,  correlations  between  the  personality 
scales  from  the  battery  and  five  pretest  performance  criteria  were  examined. 
These  correlations  appear  in  Table  4.  Because  many  of  Ghiselli's  Self- 
Description  Inventory  (SDI)  scales  correlated  near  zero  with  the  performance 


24 


Correlations  Between  Pretest  Predictor  Battery 
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19.  Decision  Making  Ability 

20.  Soclooetric  Popularity 

21.  Perceived  Maturity 


criteria,  this  instrument  was  eliminated  from  the  battery  with  the  exception 
of  a single  scale,  Self-Assurance,  which  correlated  .23  with  the  production 
index.  To  ease  administration  of  this  scale,  only  the  "most  descriptive 
adjective”  items  were  used.  In  fact,  that  half  of  the  scale  correlated 
more  highly  with  the  objective  production  index  than  did  the  total  scale 
(£  ■ .30  vs.  £ ■ .23),  suggesting  that  no  validity  would  be  lost  by  this 
change.  Also,  Dunnette's  Sales  Effectiveness  Scale,  which  has  the  same 
format  as  the  SDI,  contains  primarily  most  descriptive  adjective  pairs; 
therefore,  those  32  Sales  Effectiveness  Scale  items  with  the  least  descrip- 
tive format  were  eliminated.  Like  the  elimination  of  the  SDI  items,  this 
change  provided  no  appreciable  change  in  pretest  validity.  Altogether, 
the  SDI  and  Sales  Effectiveness  section  of  the  battery  was  reduced  from 
106  to  64  items  based  upon  pretest  information. 

Since  item-level  analyses  were  to  be  performed  on  the  Strong-Campbell 
Interest  Inventory,  it  remained  Intact  for  the  main  battery  administration 
effort. 

Additionally,  item- level  analyses  were  to  be  performed  on  the  Biographical 
and  Opinion  Survey;  consequently,  no  items  were  eliminated  at  this  time, 
though  a few  of  the  response  alternatives  were  changed  where  it  seemed 
warranted.  This  instrument  and  a sample  revision  appear  in  Appendix  D. 

Forming  Predictor-Criterion  Correlation  Hypotheses  Using  Pretest  Data 

Data  from  the  pretest,  besides  helping  to  refine  the  predictor  battery 
and  performance  criterion  materials,  aided  in  formulating  hypotheses 
about  predictor-criterion  relationships.  These  hypotheses  are  useful  in 
estimating  the  cross-validity  of  the  battery.  Essentially,  forming  judg- 
ments about  predictor-criterion  relationships  helps  to  eliminate  items  or 
scales  from  consideration  as  predictors  for  a criterion  when  their  con- 
ceptual relationships  with  that  criterion  are  weak  or  nonexistent.  For 
example,  Borman  (1973)  and  Dunnette  (1976a)  have  matched  personality, 
ability,  and  Interest  scales  to  individual  performance  criteria  according 
to  the  conceptual  ties  between  predictor  constructs  and  the  performance 
dimensions.  These  studies  employed  the  Monte  Carlo  procedure  described 
in  Appendix  E to  assess  the  empirical  validity  of  predictor  composites 
formed  by  combining  the  scales  selected. 

The  approach  used  in  the  present  study  extends  this  strategy  by  using 
both  conceptual  ties  and  empirical  data  to  help  make  judgments  about  these 
relationships.  It  estimates  the  cross-validity  of  various  parts  of  the 
battery  in  one  of  two  ways:  (1)  by  applying  the  preselected  keys  directly 
in  the  main  administration  samples;  or  (2)  by  performing  the  Monte  Carlo 
analysis.  Reasons  for  these  choices  appear  in  the  next  section. 

Accordingly,  pretest  correlations  between  predictors  and  criteria  were 
examined  closely  and  the  conceptual  links  between  each  predictor  scale  or 
Item  and  each  performance  criterion1  were  studied  to  arrive  at  the  list  of 

1The  criteria  referred  to  here  are  pretest  criteria,  not  main  sample 
criteria.  Thus , these  hypotheses  were  formed  for  the  three  performance 
criteria  generated  by  the  pretest  factor  analysis.  However,  the  criterion 
analysis  of  the  main  sample  data  indicated  substantially  the  same  underlying 
factors,  thereby  making  the  hypothesized  criterion  relationships  appropriate 
for  main  sample  data. 
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hypothesized  predictor-criterion  linkages  presented  in  Appendix  7,  Thus, 
both  a simple  cross-validation  model  and  the  Monte  Carlo  procedure  developed 
by  Bosse  and  used  by  Borman  (1973) , Dunnette  and  Motowidlo  (1975),  Rosse 
and  Hellervik  (1975),  and  Dunnette  (1976b)  were  selected  for  use  with  the 
main  sample  in  this  study. 

Assigning  Cross-Validation  Strategies  to  Different  Sections  of  the  Predictor 
Battery 

These  predictor  items  and  scales  selected  on  the  basis  of  pretest  informa- 
tion were  separated  (for  each  criterion)  into  three  predictor  sets — per- 
sonality, interest,  and  biographical.  The  next  step  was  to  decide  where 
Monte  Carlo  item  analysis  procedures  should  be  followed  and  where  simply 
cross-validating  the  preselected  keys  on  the  main  sample  was  more  appro- 
priate. 

Personality  Scales 

The  hypothesized  personality  predictors,  containing  variables  at 
the  scale  level,  did  not  appear  to  be  good  candidates  for  Monte  Carlo 
item  analysis  treatment.  The  pretest  results  indicated  that  several  per- 
sonality scales  correlated  significantly  with  one  or  more  criteria  and  that 
the  pattern  of  relationships  between  scales  and  criteria  was  conceptually 
reasonable.  Thus,  it  was  decided  to  simply  cross-validate  the  personality 
composites  as  developed  from  pretest  data  with  the  criteria  toward  which 
they  were  targeted. 

SCII 


The  Monte  Carlo  procedure  appeared  appropriate  for  investigating 
relationships  between  the  SCII  and  each  of  the  five  criteria  for  the  Navy 
sample.  This  is  because  relationships  between  SCII  scales  (i.e. , basic, 
occupational,  and  Holland  theme  scales)  and  the  criteria  in  the  pretest 
data  were  not  very  promising;  thus,  some  kind  of  item  analysis  seemed 
warranted.  And  also,  the  main  sample  size  for  the  Navy  (N  " 267)  was  large 
enough  that  the  Monte  Carlo  item  analysis  procedure  should  create  rela- 
tively stable  keys  for  each  run,  thereby  ensuring  reasonably  accurate  es- 
timates of  the  cross-validity  of  the  item  keys.  See  Appendix  E for  a 
further  description  of  the  Monte  Carlo  procedure. 

For  the  Marine  Corps  sample  (N  » 118),  the  sample  size  appeared 
to  be  too  small  to  provide  stable  keys  for  each  Monte  Carlo  run.  There- 
fore, it  seemed  preferable  to  use  a simple  cross-validation  modal  to  es- 
timate the  validity  of  keys  selected  on  the  basis  of  pretest  data.  Accord- 
ingly, for  each  criterion,  SCII  items  were  selected  and  keyed  (i.e.,  a 
direction  was  specified)  that  showed  promise,  both  conceptually  and 
empirically  (pretest  data),  for  predicting  performance.  These  keys 
were  then  cross-validated  on  the  main  sample  (N  - 118). 

Biographical  Survey 

The  relatively  small  number  pf  biographical  survey  items  selected 
for  cross-validation  suggested  that  the  Monte  Carlo  procedure  might  be 
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appropriate  for  both  samples.  Even  for  the  Marine  Corps  group,  the  small 
number  of  items  included  in  the  cross-validation  item  pools  meant  that 
relatively  few  invalid  items  would  enter  the  keys  on  individual  Monte 
Carlo  runs.  Therefore,  the  Monte  Carlo  procedure  was  expected  to  provide 
a reasonably  accurate  estimate  of  the  keys'  cross-validities. 

Description  of  the  Sample  for  the  Main  Study 

For  the  main  administration  of  the  inventory  and  performance  rating 
materials,  a geographically  representative  sample  of  267  Navy  recruiters 
from  ten  Navy  Recruiting  Districts  (NSDs)  was  selected.  A list  of  those 
NRDs  and  the  number  of  Navy  recruiters  participating  from  each  is  shown 
in  Figure  1,  which  presents  the  same  information  for  the  Marine  Corps 
sample.  While  offering  a good  spread  geographically,  neither  the  Navy 
nor  the  Marine  Corps  sample  is  representative  in  terms  of  a rural, 
suburban,  inner  city  split.  The  inner  city  and  suburban  recruiter  pop- 
ulation is  overrepresented,  though  recruiters  canvassing  in  rural  areas 
were  Included  in  the  Amarillo,  El  Paso,  and  Oklahoma  City  groups. 

A sampling  plan  which  underrepresents  persons  recruiting  in  rural 
areas  was  used  here  for  two  reasons.  First,  countrywide,  inner  city, 
and  suburban  recruiters  as  a group  comprise  the  vast  majority  of  the  re- 
cruiting force,  and  they  recruit  many  more  individuals  into  the  Navy  and 
Marine  Corps  than  do  their  counterparts  working  rural  areas.  Second,  the 
cost  of  collecting  data  would  have  been  Increased  considerably  with 
any  substantial  inclusion  of  recruiters  from  rural  stations. 

Overall,  the  samples  appear  to  be  representative  of  the  Navy  and  Marine 
Corps  suburban  and  inner  city  recruiter  populations. 

Procedures  for  the  Main  Study 

To  collect  predictor  Inventory  and  performance  ratings,  meetings  were 
arranged  with  small  groups  of  no  more  than  16  recruiters  at  a time.  This 
procedure  provided  maximum  personal  attention  to  participating  recruiters 
during  the  performance-rating  part  of  the  program.  The  small  group  sessions 
were  designed  to  facilitate  the  training  of  raters  and  to  provide  consider- 
able opportunity  for  questions  about  the  rating  form,  the  Inventory  battery, 
and  the  study  in  general. 

Each  session  began  with  a 10-  to  15-minute  briefing  that  outlined 
the  purpose  of  the  study  and  explained  the  procedures  to  be  followed  during 
the  3-hour  session.  After  the  briefing,  the  group  was  divided  into  two 
roughly  equal-sized  subgroups,  and  one  subgroup  began  the  performance  ratings 
while  the  other  subgroup  started  completing  the  inventories.  After  the 
first  subgroup  completed  the  performance  ratings,  the  recruiters  began  com- 
pleting the  trial  inventories.  The  other  subgroup  performed  the  rating  task 
after  coapleting  the  inventories. 

In  the  performance  rating  session,  participants  were  asked  to  read  the 
directions  for  the  rating  scales  (see  Appendix  C for  a copy  of  the  direc- 
tions end  of  the  reting  eceles)  end  provided  sn  example  of  how  to  rete  an 
individual  or  the  first  dimension.  Locating  and  Contacting  Qualified 
Prospects.  They  were  allowed  to  ask  questions  and  were  Instructed  to 
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Navy  Recruiters 


Marine  Corpe  Recruiters 


Area  1 

Philadelphia  28  15 

Area  3 

Southeast  region 
(includes  Jacksonville, 

Miami,  Tampa)  33  17 

Area  4 

Washington,  D.  C.  33  20 

Area  5 

Minneapolis  — 12 

Chicago  33  20 


Area  7 

Houston  26 

Southwest  (Includes 
Amarillo,  Albuquerque, 


and  El  Paso)  25  8 

Oklahoma  City  22  7 

Area  8 

Oakland  35  10 

Los  Angeles  32  9 

Total  267  118 


Figure  1.  Description  of  main  samples  according  to  Navy  Area  Recruiting 
Districts. 
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evaluate  themselves  and  their  station  mates  who  were  participating  in 
the  research  program. 

In  a separate  rating  session,  performance  evaluations  were  gathered 
from  supervisors  selected  from  the  supervisory  levels — Chief  Recruiter, 
Enlisted  Processing  Officer,  and  Zone  Supervisor  (Noncommissioned  Officer 
in  Charge  or,  in  the  Marine  Corps,  Sergeant  Major).  As  with  the  pretest, 
two  supervisory  ratings  were  gathered  for  each  recruiter  whenever  possible. 

In  the  testing  sessions,  recruiters  were  first  briefed  about  the  content 
of  the  inventories  to  be  completed,  then  asked  to  begin  responding  to  the 
trial  inventory  battery  at  their  own  speed. 

Finally,  production  statistics  were  gathered  for  recruiter  participants 
in  the  form  of  the  number  of  persons  each  had  brought  into  the  Navy  or 
Marine  Corps  from  May  through  October  of  1976.  Thus,  the  main  data 
collection  effort  yielded  self,  peer,  and  supervisory  ratings;  production 
information;  and  responses  to  the  revised  Inventory  battery. 
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RESULTS 


Criterion  Development  Results 

The  performance  ratings  gathered  from  supervisors  and  recruiters  were, 
as  in  the  pretest,  analyzed  to  evaluate  various  rating  errors.  Interrater 
reliability  of  the  ratings  was  assessed  along  with  leniency,  restriction- 
of-range,  and  halo  errors.  Then,  the  dimensionality  of  the  performance 
ratings  was  examined  in  order  to  create  a final  set  of  criteria  intended 
to  comprehensively  and  efficiently  reflect  recruiter  effectiveness. 

Interrater  Agreement 

Table  5 shows  the  intraclass  correlation  coefficients  indicating 
the  interrater  reliability  of  ratings  from  the  self,  peer,  and  supervisory 
sources  pooled  together.  Results  are  displayed  for  the  Navy  and  Marine 
Corps  samples.  As  in  the  pretest,  the  reliabilities  of  ratings  from  all 
combinations  of  these  sources  were  examined  and  the  peer-supervisor  and 
self-peer-supervisor  groupings  were  foind  to  provide  the  most  reliable 
mean  ratings.  Hie  self-peer-supervisor  grouping  was  selected  because 
the  factor  structure  of  ratings  from  this  combination  of  sources  was  more 
readily  lnterpretable  than  th&  factor  structure  obtained  from  the  peer- 
supervisor  ratings.  In  general,  reliability  is  acceptable  for  the  Navy 
(median  _r  - .57)  but  marginal  for  the  Marine  Corps  (median  _r  » .48)  scales. 

Leniency 

As  was  explained  earlier,  the  rating  errors  of  leniency,  restric- 
tion of  range,  and  halo  can  be  evaluated  only  indirectly.  As  for  leniency, 
the  mean  ratings  on  individual  dimensions  in  the  Navy  sample  range  from 
6.86  to  7.81  on  the  10-point  behavior  dimensions  and  from  6.23  to  7.01 
on  the  9-point  supplementary  scales.  For  the  Marine  Corps  sample,  the 
ranges  are  6.55  to  7.83  on  the  10-point  and  5.79  to  6.92  on  the  9-point 
scales.  See  Table  6 for  a complete  presentation  of  the  means.  These  mean 
ratings  are  far  enough  from  the  scales'  highest  points  that  leniency  does 
not  appear  to  be  a serious  problem. 

Restriction  of  Range 

The  standard  deviations  of  the  ratings  for  each  dimension  are  also 
depicted  in  Table  6.  They  indicate  that  a reasonable  degree  of  variation 
in  effectiveness  is  present  in  the  rated  performance  of  the  recruiters  in 
the  two  samples. 

Halo 


The  magnitude  of  the  correlations  between  dimensions  is  in  general 
high.  In  the  combined  Navy  and  Marine  Corps  sample,  for  example,  these  cor- 
relations range  from  .35  to  .81  for  the  ratings  derived  from  the  three 
sources  pooled  together.  The  median  interdimension  correlation  is  .62. 

The  magnitude  of  these  relationships  suggests  that  some  degree  of  halo  error 
is  present  in  the  ratings. 
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Interrater  Agreement  Indices  for  Self,  Peer,  and  Supervisor  Ratings 
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Determining  Dimensionality  of  the  Ratings 


Although  halo  did  appear  to  pervade  the  ratings,  an  effort  was  made 
to  identify  factors  that  might  underlie  the  relationship  between  dimensions 
The  Navy  and  Marine  Corps  ratings  were  pooled  and  a principal-components 
factor  analysis  was  applied  to  the  intercorrelations  among  dimensions  for 
each  of  the  following  rating  source  groupings:  (1)  peer-supervisor;  and 
(2)  self-peer-supervisor.  The  eigenvalues  of  the  two  solutions  were 
examined  and  the  two-,  three-,  and  four-factor,  varlmax-rotated  solutions 
were  studied  for  their  psychological  meaningfulness.  The  three-factor 
solution  for  the  self-peer-supervisor  ratings  appeared  most  meaningful. 

That  solution  appears  in  Table  7.  The  following  factors  were  named: 

I.  Selling  Skills:  selling  the  Navy  or  Marine  Corps  effectively 
to  prospects;  displaying  confidence  and  effectiveness  in  the  recruiting 
sequence — prospecting,  selling,  and  closing. 

II.  Human  Relations  Skills:  establishing  and  maintaining  good 
interpersonal  relations  with  prospects  and  recruits,  persons  in  the  com- 
munity, fellow  recruiters,  and  supervisory  personnel. 

III.  Administrative  Skills:  planning  ahead  and  organizing  time 
efficiently;  completing  paperwork  accurately  and  on  time. 

The  content  of  these  factors  agrees  closely  with  the  content  of  the 
pretest  factors,  suggesting  that  the  three-factor  solution  represents  a 
stable  picture  of  the  underlying  dimensions  of  performance  on  the  Navy  or 
Marine  Corps  recruiter  job.  Further,  the  main  Navy  and  the  Marine  Corps 
8 ample  ratings  were  analyzed  separately  to  evaluate  the  comparability  in 
factor  structure  across  two  groups.  The  two  three-factor  solutions  were 
markedly  similar  when  each  was  interpreted  by  itself.  In  addition,  a crude 
numerical  index  of  the  comparability  in  these  factor  solutions  was  provided 
by  correlating  the  factor  loadings  for  the  three  pairs  of  varlmax-rotated 
factors.  Those  correlations  were  .83,  .69,  and  .86,  suggesting  further 
the  stability  of  these  orthogonal  dimensions.  Finally,  the  reliabilities 
of  the  factor  scores  are  reasonably  high.  Table  8 indicates  that  these 
reliabilities  range  from  .51  to  .68  for  the  two  samples. 

In  summary,  based  on  conceptual  considerations,  reliability  results 
and  results  of  the  factor  analyses,  all  ratings — self,  peer,  and  supervisor 
ratings — were  selected  for  subsequent  steps  in  the  study. 

Developing  the  Objective  Production  Index 

As  in  the  pretest  study,  monthly  production  data  were  available 
for  many  recruiters  in  the  sample  for  the  6 months  from  May  through 
October  of  1976  (i.e.,  number  of  accessions  for  each  of  the  6 months). 

The  stability  of  the  corrected  production  index,  adjusted  within  district 
in  the  same  way  as  was  pretest  production,  was  evaluated  by  computing 
its  reliability  using  methods  Identical  to  those  employed  with  pretest 
data.  The  Intraclass  correlation  coefficients  representing  the  reli- 
ability of  the  mean  corrected  production  scores  across  the  6 months  were 
.73  for  the  Navy  sample  and  .83  for  the  Marine  Corps  sample.  These 
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Three-Factor  Varimax  Rotated  Solution  for  Self-Peer-Supervisor 
Ratings  Pooled  Together:  Navy  and  Marine  Corps  Main  Sample  Combined 
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Table  8 


Int ercor relations  and  Reliabilities  of  the  Five  Criteria 


Int  ercorrelat ions 

Reliabilities 

Criteria 

1 

2 

3 

4 

5 

Navy 

.68a 

1. 

Selling  Skills 

— 

.53 

2. 

Human  Relations  Skills 

.23 

— 

.62 

3. 

Administrative  Skills 

.21 

.12 

— 

.78 

4. 

Overall  Performance 

.79 

.50 

.51 

— 

.73 

5. 

Production 

.43 

.15 

.17 

.52 

— 

Marine  Corps 

.63 

1. 

Selling  Skills 

— 

.55 

2. 

Human  Relations  Skills 

.03 

— 

.51 

3. 

Administrative  Skills 

.11 

-.02 

— 

.66 

4. 

Overall  Performance 

.72 

.42 

.40 

— 

.83 

5. 

Product ion 

.45 

.31 

.18 

.59 

— 

Reliabilities  for  the  Selling,  Human  Relations,  and  Administrative  Skills 
dimensions  Index  the  stability  of  the  factor  scores  associated  with  these 
dimensions. 
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relatively  high  reliabilities  indicate  that  the  corrected  accessions-per- 
month  measure  is  at  least  consistent  over  time;  recruiters  who  have  high 
production  one  month  tend  to  have  high  production  during  other  months, 
and  recruiters  less  productive  in  a given  month  tend  to  be  relatively 
unproductive  during  other  months  as  well.  As  discussed  earlier,  high 
reliability  implies  nothing  about  this  measure's  relevance  as  an  index 
of  job  effectiveness,  but  the  index  does  possess  a certain  face  validity 
as  an  indicator  of  a recruiter's  "bottom  line'1  contribution  to  the  Navy 
or  Marine  Corps  recruiting  effort. 

Performance  Criteria 


The  production  index,  along  with  the  scores  on  these  three  factors 
identified  by  analyses,  provided  four  performance  criteria.  As  in  the  pre- 
test, a fifth  criterion — Overall  Performance — was  added  to  serve  as  a summary 
criterion. 

Relationships  Between  Criteria 

Table  8 depicts  the  correlations  between  the  five  criteria  for  the 
Navy  and  Marine  Corps  separately.  As  expected,  the  first  three  criteria 
are  relatively  uncorrelated  because  of  the  manner  in  which  the  factor 
scores  were  computed.  As  such,  they  represent  effectiveness  criteria  in 
three  distinctly  different  aspects  of  the  Navy  and  Marine  Corps  recruiter 
job.  The  Overall  Performance  rating  correlates  moderately  with  the  Human 
Relations  Skills  and  Administrative  Skills  criteria,  and  highly  (rs  - .79 
and  .72)  with  Selling  Skills.  This  pattern  of  correlations  appears  reason- 
able because  the  Selling  Skills  criterion  defines  the  heart  of  the  recruiting 
sequence — prospecting,  selling,  and  closing — and  therefore  should  be  the 
criterion  most  closely  related  to  overall  performance.  The  same  observa- 
tion holds  for  the  Production  criterion.  Of  the  three  performance  criteria 
associated  with  Individual  facets  of  the  job,  it  relates  most  highly  to 
Selling  Skills.  Further,  the  correlations  between  this  measure  and  the 
Overall  Performance  rating  is  at  least  moderate  (rs  - .52  and  .59)  and  is 
higher  than  the  relationships  between  production  and  performance  in  any 
single  facet  of  the  job. 

Thus,  the  five  criteria  possess  adequate  to  high  reliability,  the 
criteria  measuring  performance  in  individual  facets  of  the  job  appear  to 
reflect  well  three  relatively  Independent  aspects  of  Navy  or  Marine  Corps 
recruiter  job  performance,  and  the  pattern  of  correlations  between  the 
facet  criteria  and  the  two  measures  of  overall  effectiveness  make  good 
conceptual  sense.  These  criterion  development  results  pave  the  way  for 
attempts  to  predict  performance  on  each  criterion  measure  using  informa- 
tion on  differences  between  recruiters  that  is  available  from  predictor 
data. 

Trial  Predictor  Battery  Validation  Results 

This  section  describes  relationships  between  the  five  performance 
criteria  and  predictor  composites  developed  according  to  the  hypotheses 
about  predictor-criterion  linkages.  The  composites  can  be  found  in  Appen- 
dix F.  Additionally,  it  presents  data  that  can  be  used  to  evaluate  racial 
bias  and  the  differential  validity  of  these  predictor  composites. 
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First,  Tables  9 and  10  show  the  correlations  between  personality  scales 
and  each  of  the  five  criteria  for  the  Navy  (N  - 267)  and  the  Marine  Corps 
(N  ■ 118)  separately.  For  the  Navy  sample,  the  relationships  are  not  as 
high  in  general  as  was  found  when  pretest  data  were  analyzed.  Marine 
Corps  results  are  similar  (both  in  magnitude  and  pattern  of  the  correla- 
tions) to  the  results  obtained  from  the  pretest  data. 

Cross-Validation  Results  for  Preselected  Keys 

Results  of  the  personality  composite  cross-validation  are  shown 
in  Table  11.  Administrative  Skills  is  clearly  the  criterion  most  readily 
predicted  by  the  personality  composites  for  both  the  Navy  and  the  Marine 
Corps  samples.  The  Production  criterion,  though  reasonably  well  predicted 
in  the  Marine  Corps  sample,  correlated  near  zero  with  the  Production  com- 
posite within  the  Navy  group. 

Table  12  displays  results  of  correlations  between  each  of  the  five 
criteria  and  the  SCI1  keys  for  the  Marine  Corps  groups.  The  keys  were 
formed  on  the  basis  of  pretest  Information.  These  cross-validity  coeffici- 
ents hover  about  the  .20  to  .25  level  with  the  exception  of  the  relation- 
ship between  Production  and  the  Production  SCII  predictor  key,  for  which 
t_  - .10. 

Monte  Carlo  Cross-Val ldation  Results 


For  the  Monte  Carlo  analyses,  only  those  items  preselected  on  the 
basis  of  pretest  data  and  conceptual  considerations  were  included  in  the 
pool  that  entered  Monte  Carlo  runs.  Further,  the  direction  (l.e.,  positive 
or  negative)  of  many  of  the  relationships  was  specified  ahead  of  time. 
Identifying  the  intended  direction  prevents  an  item  from  entering  a Monte 
Carlo  run  when  it  meets  the  item  inclusion  criterion  in  the  wrong  direction. 
The  direction  and  criterion  limit  options  of  the  Monte  Carlo  computer  pro- 
gram are  explained  more  fully  in  Appendix  E. 

Results  of  the  Monte  Carlo  analyses  for  the  SCII  (Navy  sample) 
appear  in  Table  13.  The  "total  sample"  validity  coefficients  represent  the 
correlation  between  each  criterion  and  the  keys  formed  from  total  sample 
data  (N  ■ 267).  The  cross-validity  coefficients  are  median  Monte  Carlo 
estimates  of  these  keys'  validities.  They  are  probably  underestimates  of 
the  validities  because  the  keys  developed  for  each  Monte  Carlo  run  are 
formed  on  only  half  the  total  sample  and  are  therefore  likely  to  be  less 
stable  than  the  final  keys  developed  using  the  entire  sample. 

Table  14  depicts  Monte  Carlo  results  for  the  Biographical  Survey 
data.  In  general,  the  results  are  disappointing.  Only  with  the  criteria 
Administrative  Skills  and  Overall  Performance  in  the  Navy  sample  do  cross- 
validities  reach  even  the  .20s.  For  the  Marine  Corps  groups,  the  validi- 
ties are  only  .15.  Again,  these  Monte  Carlo  validities  are  probably  under- 
estimates of  the  final  keys'  true  validities. 
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Table  10 

Correlations  Between  Predictor  Battery  Personality  Scales 
and  the  Five  Performance  Criteria:  Main  Sample,  Marine  Corps 


AO 


Table  11 


Cross-Validation  Results  for  the  Personality  Composites 


Criteria 

Navy 

Marine  Corps 

Selling  Skills 

.21** 

.19* 

Human  Relations  Skills 

.16** 

.19* 

Administrative  Skills 

.29** 

.38** 

Overall  Performance 

.21** 

.24** 

Production 

.01 

.24* 

*p  < .05 

**p  < .01 

Table  12 

Cross-Validation  Results  for  the 

(Marine  Corps  Only) 

SCII  Keys 

Criteria 

Marine  Corps 

Selling  Skills 

.19* 

Human  Relations  Skills 

.24** 

Administrative  Skills 

.20* 

Overall  Performance 

.26** 

Production 

.10 

aSCIl  results  for  the  Navy  were  obtained  using  the  Monte  Carlo  cross- 
validation  procedure.  Those  results  are  contained  in  Table  13. 

*p  < .05 

**p  < .01 
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Table  13 


Monte  Carlo  Results — SCII,  Navy  Sample 

Total  Sample 

Cross-Validity 

Criteria 

Validity  Coefficients 

Coefficients 

Selling  Skills 

.32** 

.19** 

Human  Relations  Skills 

.23** 

.17** 

Administrative  Skills 

.38** 

.26** 

Overall  Performance 

.33** 

.17** 

Production 

.11 

-.02 

**p  < .01 

Table  14 

Monte  Carlo  Results— Biographical  Survey 

Navy 

Marine  Corps 

Total  Sample 

CrosB- 

Total  Sample 

Cross- 

Validity 

Validity 

Validity 

Validity 

Criteria 

Coefficients 

Coefficients 

Coefficients 

Coefficients 

Selling  Skills 

,35** 

.16** 

.40** 

-.01 

Human  Relations 

Skills 

.30** 

.05 

.46** 

.10 

Administrative 

Skills 

.42** 

.23** 

.49** 

.15 

Overall 

Performance 

.39** 

.23** 

.53** 

.15 

Production 

.30** 

.01 

.46** 

.08 

**p  < .01 


A 
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Estimated  Validity  for  the  Final  Composites  Pooled  Together 


To  estimate  the  validity  of  the  various  predictor  composites  taken 
together,  a simple  procedure  was  employed  that  uses  the  cross-validity 
coefficients  and  intercorrelations  between  the  predictor  composites. 

The  following  formula  was  applied: 

V - E w.r.  / / E I w.w  r,, 

t i-i  1 W / i j 1 J « 

where  - estimated  cross-validity  of  the  combined  com- 

posites, 

r - estimated  cross-validity  of  the  ith  predictor 
y composite  with  the  criterion  y (i  ■ 1,  2,  . . 
m) , 

w^  and  Wj  - the  weights  applied,  and 

r.  - the  intercorrelation  of  the  ith  and  jth  predic- 
J tor  composites. 


As  weights,  the  standard  deviations  of  composite  scores  were  used. 
This  procedure  has  the  effect  of  giving  extra  weight  to  longer,  and  presum- 
ably more  reliable,  composites.  Also,  composites  were  eliminated  from 
further  consideration  in  this  analysis  when  their  validity  did  not  appear 
sufficient  to  warrant  Inclusion  in  the  final  set  of  predictor  measures 
(see  Table  15  for  the  composites  included). 

Thus,  the  inventory  responses  for  persons  in  the  two  samples.  Navy 
(main  sample)  and  Marine  Corps,  were  scored  according  to  the  final  keys 
developed  for  personality,  SCII,  and  biographical  composites.  Then  the 
composite  scores  were  lntercorrelated  and  these  intercorrelations,  along 
with  the  cross-validities,  were  employed  to  estimate  the  validity  of  the 
pooled  predictor  measures. 

Results  of  this  analysis  appear  in  Table  15.  Except  for  the 
Human  Relations  Skills  and  Production  criteria  in  the  Navy  sample,  es- 
timated overall  validities  are  uniformly  in  the  .20s  and  .30s. 

Black-White  Comparisons:  An  Examination  of  Bias  and  Differential 
Validity  in  the  Navy  Sample 

The  Navy  sample  contained  a large  enough  group  of  Black  recruiters 
(N  ■ 36)  that  an  assessment  of  various  black-white  racial  differences 
on  the  predictors,  criteria,  and  predictor-criterion  relationships  was 
deemed  technically  feasible.  (An  N of  30  has  been  offered  as  a minimum 
sample  size  for  validation  research-- Federal  Register,  1976).  The 
Marine  Corps  sample  contained  only  21  black  recruiters,  rendering  any 
such  comparisons  technically  inappropriate.  Neither  the  Navy  nor  the 
Marine  Corps  samples  contained  sufficiently  large  numbers  of  women, 
Mexlcan-Amerlcane,  or  any  other  minority  group  to  make  other  kinds 
of  intergroup  comparisons  possible. 
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Int ercorre lac  ions  of  Predictor  Composites/Keys  and 
Estimated  Validity  for  the  Final  Composites  Pooled  Togethe 
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Table  16  shows  the  means  and  standard  deviations  on  predictor  com- 
posites and  on  the  criteria  for  white  (N  ■ 197)  and  black  (IJ  - 36)  Navy 
recruiters  separately.  These  results  suggest  that  mean  predictor  scores 
for  the  black  and  white  recruiters  in  the  sample  are  very  similar.  The 
largest  differences  occur  for  the  Selling  Skills  biographical  key,  on 
which  black  Navy  recruiters  score  approximately  one-third  of  a standard 
deviation  higher  than  their  white  counterparts.  Differences  in  means  or 
standard  deviations  are  very  slight  in  other  cases.  Except  for  Administra- 
tive Skills,  black  recruiters  are  rated  as  a little  more  effective  than 
white  recruiters,  but  again,  the  differences  are  not  great — one-fourth 
standard  deviation  at  the  most. 


Table  16 

Black-White  Differences:  Means  and  Standard  Deviations 
on  Predictor  Composites  and  Performance  Criteria 


Criteria 

Selling  Skills 

Human  Relations 
Skills 

Administrative 

Skills 

Overall 

Performance 

Variables 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

Personality 

Black 

205.26 

26.06 

365.66 

59.44 

312.94 

51.59 

308.71 

34.97 

White 

202.26 

26.38 

361.95 

63.46 

305.00 

50.84 

305.11 

39.34 

sen 

Black 

22.33 

16.69 

20,78 

20.66 

5.61 

8.91 

21.22 

17.67 

White 

21.78 

15.36 

13.42 

20.74 

7.87 

8.75 

19.35 

15.48 

Biographical 

Black 

-.17 

1.11 

a 

a 

-.23 

2.22 

-.66 

2.11 

White 

-.58 

1.32 

A 

a 

-.  61 

1.78 

-1.09 

1.98 

Actual  Performance 

Black  51.04b 

52.01 

50.96 

6. 76b 

White 

48.38 

— 

49.86 

— 

5-1.74 

— 

6.39 

— 

aBlographical  items  did  not  yield  a scale  for  Human  Relations  Skills. 

bThe  mean  performance  scores  are  T-scores  (mean  ■ 50,  SD  - 10)  for  all 
criteria  except  Overall  Performance.  Mean  scores  for  that  criterion  are 
mean  ratings  on  the  Overall  Performance  dimension. 


Table  17  shows  the  validity  coefficients  for  the  black  and  white 
recruiter  groups  separately.  As  with  the  means  and  standard  deviations, 
few  substantial  differences  can  be  noted  between  the  two  samples  even  though 
the  small  size  of  the  black  group  makes  the  validity  coefficients  for  the 
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black  sample  quite  unstable.  Only  Selling  Skills  shows  consistent  differ- 
ences, with  the  validity  higher  in  the  white  recruiter  group.  However, 
even  these  differences  are  not  statistically  significant  and  are  not  par- 
ticularly striking. 


Table  17 

Black-White  Differences:  Validity  Coefficients 
for  Final  Predictor  Composites 


Predictor 

Selling 

Skills 

Human  Relations 
Skills 

Adminlstrat ive 
Skills 

Overall 

Performance 

Blacks 

Whites 

Blacks 

Whites 

Blacks 

Whites 

Blacks 

Whites 

Personality 

.07 

.21 

.28 

.16 

.44 

.16 

.32 

.20 

SCIIa 

.16 

.35 

.19 

.25 

.33 

.38 

.32 

.34 

Biographical 

.07 

.37 

— 

— 

.59 

.38 

.40 

.39 

aValidity  coefficients 

for  the 

SCII  and 

Biographical  Survey 

are  total 

sample 

correlations,  not  cross-validities. 


Overall,  these  data  suggest  that  the  final  predictor  composites  will 
not  show  bias  against  either  white  or  black  candidates  for  the  Navy  re- 
cruiter Job. 


CONCLUSIONS 


Performance  Measurement:  Differences  Between  the  Present  Study  and 
Previous  Efforts 


During  this  project,  considerable  effort  went  into  the  development  of 
criteria  for  measuring  Navy  and  Marine  Corps  recruiter  effectiveness. 
Briefly,  the  investigation  of  performance  criteria  Included  studying 

(1)  various  combinations  of  rating  sources  (i.e.,  self,  peer,  and  super- 
visors), (2)  the  dimensionality  of  military  recruiter  effectiveness,  and 
(3)  the  relationship  between  recruiter  production  (i.e.,  number  of  persons 
brought  into  the  Navy  or  Marine  Corps)  and  performance  in  the  various  as- 
pects of  the  job.  These  steps  were  taken  in  order  to  arrive  at  final  per- 
formance criteria  that  reflected  relevant,  reliable,  comprehensive,  and 
yet  parsimonious  measurement  of  recruiter  performance.  The  relatively 
elaborate  criterion  development  procedures  may  well  have  led  to  the  reason- 
ably good  validities  obtained  in  the  study. 

Reviewing  briefly  the  criterion  development  approaches  taken  in  other 
studies  of  the  military  recruiter  Job  should  highlight  the  differences 
between  those  approaches  and  the  one  pursued  here.  The  three  types  of 
measures  used  in  these  previous  studies  are  (1)  uncorrected  production; 

(2)  adjusted  or  corrected  production;  (3)  ratings,  ranking,  or  nominations. 
Yet,  each  of  these  methods  may  fail  to  describe  accurately  a military  re- 
cruiter’s true  effectiveness. 

First,  production,  though  it  seems  so  legitimate  as  a "bottom  line” 
index  of  a recruiter's  effectiveness,  is  probably  riddled  with  the  kinds 
of  errors  that  plague  most  objective  Indicators  of  success.  Perhaps 
the  most  serious  potential  error  in  production  scores  is  criterion  con- 
tamination. Contamination  occurs  when  job  incumbents'  production  is  por- 
trayed so  that  some  appear  better  than  their  individual  contributions 
warrant,  while  others  appear  poorer.  Contamination  is  likely  to  be  present 
whenever  situational  factors  affect  criterion  scores  in  a stable  way.  It 
is  important  to  note  that  such  biasing  effects  are  not  detectable  from 
reliability  data.  Thus,  in  the  context  of  the  present  study,  high  reli- 
ability of  the  production  "scores"  says  nothing  about  possible  criterion 
contamination  in  these  scores. 

In  this  study  and  in  some  others  (Larriva,  1975;  Brown,  Wood,  & Harris, 
1975),  raw  production  statistics  have  been  adjusted  in  various  ways  to 
reduce  contaminating  influences.  However,  it  may  be  very  difficult  to 
partial  out  all  or  even  a large  proportion  of  these  biasing  influences. 

For  one  thing,  an  adjustment  that  equalizes  mean  production  across  districts 
or  stations  assumes  random  assignment  in  terms  of  effectiveness  to  districts 
and  stations.  This  is  unlikely  in  an  operational  setting.  Thus,  perhaps 
the  reason  that  many  studies  have  failed  to  find  substantial  relationships 
between  predictor  measures  and  production  or  adjusted  production  relates 
to  criterion  measurement  problems.  Such  difficulties  would  restrict  the 
magnitude  of  any  relationship  between  predictors  and  performance. 
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Of  course,  performance  ratings  are  fraught  with  their  own  problems. 

These  difficulties  have  been  well-documented  elsewhere  (e.g. , Campbell, 
Dunnette,  Lawler,  & Weick,  1970;  Whister  & Harper,  1962),  but  particular 
pitfalls  may  be  present  in  some  of  the  kinds  of  ratings  used  in  studies 
reviewed  in  a previous  section  of  this  report.  For  example,  ratings, 
rankings,  or  nominations  may  be  assigned  by  a commanding  officer  primarily 
on  a reputation  rather  than  on  good  knowledge  of  recruiters'  day-to-day 
job  activities.  Also,  rating  forms  utilizing  dimensions  that  are  not 
well  conceived  or  well  defined  may  cause  commanding  officers  or  others 
providing  the  criterion  ratings  to  make  imprecise  evaluations  of  recruiters. 

On  the  other  hand,  careful  attention  paid  to  defining  performance 
dimensions  and  to  selecting  the  proper  persons  to  provide  ratings  may  make 
"subjective"  performance  evaluations  reasonable  estimates  of  recruiters' 
true  effectiveness,  uncontaminated  by  situational  constraints.  Referring 
to  Dunnette 's  (1963)  modified  model  for  test  validation  and  selection 
research  (see  Figure  2)  shows  that  in  the  present  study  predictor  measures 
are  probably  being  evaluated  against  lob  behaviors  as  rated  by  supervisors 
and  recruiters.  That  is,  with  the  behavior-oriented  rating  scales  especially, 
raters  are  asked  to  identify  and  to  evaluate  behaviors  that  recruiters 
exhibit  in  performing  on  the  job.  Such  judgments  are  a step  removed  from 
the  consequences  in  Dunnette 'b  model.  Because  they  are  evaluations  of 
behavior  rather  than  of  consequences,  such  ratings  may  avoid  situational 
influences  on  performance  and  thus  should  be,  in  this  sense,  better  criteria 
against  which  to  validate  predictor  measures. 

Using  a different  conceptual  framework  to  make  a similar  argument, 
the  ratings  gathered  in  the  present  study  reflect  performance  more  than 
effectiveness  (Campbell,  Dunnette,  Arvey,  & Hellervlk,  1973).  Campbell 
et  al.  define  performance  as  job  behavior  that  can  be  evaluated  in  terms 
of  its  contribution  to  organizational  goals.  Effectiveness,  as  they  de- 
fine it,  refers  to  organizational  outcomes  (i.e.,  "consequences"  in  Figure 
2)  for  which  an  individual  may  be  only  partially  responsible.  Again, 
for  test  validation  purposes,  the  measurement  of  performance  (i.e.,  be- 
havior that  can  be  evaluated,  rather  than  outcome  indicators  such  as 
those  Campbell  et  al.  term  "effectiveness")  appears  most  conceptually 
reasonable. 

For  these  reasons,  carefully  developed  rating  scales  and  well-placed, 
well-prepared  raters  may  provide  the  best  estimates  of  military  recruiter 
performance.  In  the  present  study,  self,  peer,  and  supervisory  ratings 
proved  to  be  reasonably  reliable  and,  as  criteria,  they  provided  consider- 
ably higher  validities  than  an  adjusted  production  index  did. 

Relationships  Between  Predictor  Composites  and  Recruiter  Performance: 

Soma  Perspective 

Estimated  cross-validities  for  the  predictor  composites  developed  for 
Sales,  Human  Relations,  and  Administrative  Skills  and  for  Overall  Per- 
formance are  all  significantly  different  from  zero  at  the  .01  level  for 
the  Navy  sample  and  at  the  .05  level  for  the  Marine  Corps  sample . Also, 
the  methods  used  for  estimating  these  cross-validities  are  conservative; 
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JOB  CONSEQUENCES 

PREDICTORS  INDIVIDUALS  BEHAVIORS  SITUATIONS  (Related  to  Organizational  Goals) 


the  actual  validities  may  be  higher.  Moreover,  the  pattern  of  relation- 
ships between  predictors  and  criteria  is  conceptually  reasonable,  suggesting 
stable,  systematic  linkages  between  these  predictor  composites  and  recruiter 
performance.  Nevertheless,  one  must  still  ask  how  useful  the  predictor 
composites  are  likely  to  be  in  practice. 

One  way  of  looking  at  the  utility  of  the  predictor  composites  is  to 
employ  the  Taylor-Russell  Tables  (Taylor  & Russell,  1939)  to  assess  pro- 
jected gains  in  the  levels  of  performance  expected  from  individuals  selected 
according  to  the  new  set  of  predictors.  Given  a selection  ratio  of  .50 
(i.e.,  half  of  the  persons  applying  for  or  being  considered  for  recruit- 
ing duty  actually  become  recruiters),  a validity  coefficient  of  .20  means 
that  56  percent  of  the  persons  selected  for  recruiting  duty  can  be  ex- 
pected to  be  above  average  using  the  present  levels  of  recruiter  effec- 
tiveness as  a standard.  A validity  coefficient  of  .25  suggests  that  58 
percent  of  the  persons  selected  using  the  predictor  battery  can  be  expected 
to  achieve  an  above-average  performance.  The  corresponding  figures  for 
validity  coefficients  of  .30,  .35,  and  .40  are  60,  61,  and  63  percent 
respectively.  Thus,  given  the  magnitude  of  the  validity  obtained  for  the 
predictor  composites,  and  remembering  that  these  validities  are  probably 
underestimates,  it  can  be  projected  that  the  percent  increase  in  recruiters 
performing  above  the  present  average  would  be  5 to  12  percent  (depending 
on  the  performance  criterion)  if  the  recommended  predictor  composites  are 
used  to  select  individuals  for  recruiting  duty. 

Comparison  of  16PF  and  Predictor  Composite  Validity  Results 

Since  validity  results  were  obtained  in  this  study  for  both  the  16PF 
(see  Appendix  A)  and  a newly  developed  predictor  battery,  the  question 
arises  as  to  which  of  the  two  screening  devices  performed  better. 
Unfortunately,  results  of  this  study  cannot  answer  that  question  con- 
clusively. 

First,  the  administration  of  the  two  predictor  measures  occurred  at 
different  times  in  the  careers  of  persons  in  our  sample.  The  16PF  was 
administered  to  members  of  our  Navy  sample  either  before  selection  as 
recruiters  or  after  selection  but  during  the  training  program  before 
they  went  on  recruiting  duty.  The  predictors  developed  in  the  present 
study  were  of  course  administered  to  incumbents  who  had  served  some  time 
on  recruiting  duty.  Thus,  data  for  the  16PF  validity  analysis  conform 
more  closely  to  the  predltive  validity  model  than  do  data  for  the  pre- 
dictor composite  analysis.  This  means  that  validity  coefficients  as- 
sociated with  the  16PF  may  provide  more  faithful  estimates  of  the  validity 
of  real  interest  than  is  provided  by  the  predictor  composite  concurrent 
validity  study  results. 

On  the  other  hand,  neither  the  predictor  conposites  developed  in  the 
present  study  nor  the  performance  measures  correlated  highly  with  tenure. 
Therefore,  criterion-predictor  composite  relationships  are  not  unduly  af- 
fected by  amount  of  time  on  the  job,  which  in  turn  indicates  that  con- 
current cross-validity  estimates  may  be  good  estimates  of  the  composites' 
predictive  ability. 
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Still,  it  is  obviously  difficult  to  compare  directly  the  results  of 
the  two  analyses.  The  general  magnitude  of  correlations  with  performance 
criteria  is  higher  for  the  predictor  composites  than  for  the  16PF  composite, 
especially  for  the  Human  Relations  Skills,  Administrative  Skills,  and  Over- 
all Performance  criteria.  However,  the  16PF  composite  does  correlate  .21 
with  Selling  Skills  and  .19  with  Production,  and  several  individual  16PF 
scales  correlate  in  the  .20s  with  performance  criteria.  These  results 
suggest  at  least  a modicum  of  validity  (see  Appendix  A for  these  results). 
Ideally,  both  sets  of  measures  should  be  studied  in  a predictive  validity 
framework  and  the  relationships  between  these  measures  and  performance 
examined  to  assess  their  validity.  At  present,  it  can  only  be  concluded 
that  the  16PF  possesses  some  validity  for  predicting  Selling  Skills,  Over- 
all Performance,  and  Production,  and  that  our  predictor  battery  shows  good 
promise  for  predicting  Navy  recruiter  success,  most  notably  in  the  Selling 
and  Administrative  Skills  areas.  The  predictor  battery  developed  in  this 
study  shows  even  more  promise  for  predicting  Marine  Corps  recruiter  effec- 
tiveness, particularly  for  the  Administrative  Skills  and  Overall  Performance 
criteria. 


RECOMMENDATIONS 


Studying  the  Predictive  Validity  of  the  Predictor  Composite 

Cross-validity  coefficients  computed  in  the  present  study  are  only 
estimates  of  the  composites’  predictive  validity.  These  estimates  appear 
to  be  reasonable,  even  conservative,  but  the  only  way  to  ensure  that  they 
are  is  to  perform  a predictive  study.  Thus,  paper-and-pencil  measures 
consisting  of  those  parts  of  the  predictor  battery  found  to  be  valid  in 
the  present  study  should  be  administered  to  a sample  of  enlisted  personnel 
who  are  being  considered  for  recruiting  duty.  (See  page  54  for  a more  ideal 
approach  to  developing  an  improved  predictor  battery. ) These  persons  should 
be  led  to  believe  that  scores  on  the  battery  might  be  used  for  selection 
purposes,  although,  according  to  this  plan,  scores  on  the  battery  would 
not  affect  selection  decisions.  Presumably,  some  of  the  persons  taking 
the  battery  would  subsequently  be  placed  in  recruiting  billets.  After  6 
months  to  a year  on  recruiting  duty,  performance  information  could  be 
gathered  for  those  persons,  and  the  relationship  between  their  performance 
and  scores  on  the  predictor  composites  evaluated. 

As  a head  start  on  such  a project,  the  Navy  Recruiting  Command  is  ad- 
ministering the  battery  containing  the  predictor  composites  to  200  persons 
being  considered  for  recruiting  assignments.  Thus,  with  minimal  additional 
effort,  merely  continuing  to  administer  the  battery  to  appropriate  Fleet 
personnel  and  gathering  performance  ratings  after  an  adequate  number  of 
these  persons  have  served  in  a recruiting  billet  for  6 to  12  months,  the 
Navy  can  determine  the  predictive  validity  of  the  battery  developed  in  this 
study. 

Studying  the  "Fakabillty"  of  the  Predictor  Battery 

The  present  study  has  suggested  that  several  items  and  scales  from 
the  personality,  interest,  and  biographical  domains  show  reasonably  high 
validity  as  indicators  of  recruiter  performance.  However,  the  administra- 
tive set  for  persons  in  our  study  may  have  been  different  from  the  kind  of 
set  they  would  have  had  if  they  were  taking  the  inventories  as  applicants 
before  being  accepted  for  recruiting  duty.  Persons  completing  a test  or 
Inventory  before  they  are  hired  may  try  to  respond  to  items  as  they  think 
the  selection  decision-makers  might  want  them  to,  while  the  recruiters 
in  our  concurrent  validity  sample  were  less  likely  to  respond  in  that 
manner  because  they  had  nothing  to  gain  from  looking  good. 

Fortunately,  at  least  two  research  studies  (Dunnette,  McCartney,  Carlson, 
& Kirchner,  1962;  Abrahams,  Neumann,  & Glthens,  1971)  suggest  that  persons 
taking  personality  or  interest  inventories  in  an  actual  employment  setting 
tend  to  respond  much  more  honestly,  i.e.,  fake  less,  than  persons  who  are 
Instructed  to  fake  their  responses.  In  fact,  the  IXinnette  et  al.  study 
indicates  that  the  mean  scores  of  a group  applying  for  employment  are 
very  similar  to  those  of  a group  tested  only  for  research. 

Thus,  responses  made  by  our  concurrent  validity  group  (research  set) 
will  probably  be  similar  to  those  of  persons  who  are  being  considered  for 
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recruiting  duty.  However,  the  only  way  to  ensure  that  the  predictors  are 
not  fakable  in  this  sense  is  to  perform  an  empirical  test  of  the  hypothesis. 
The  predictor  data  referred  to  earlier  can  also  be  used  to  evaluate  this 
kind  of  fakability.  Mean  responses  of  the  concurrent  validity  sample 
can  be  compared  to  the  mean  scores  obtained  by  the  200  persons  taking  the 
same  inventories  in  a selection  setting.  Those  items  and  scales  showing 
good  validity  and  small  differences  in  scores  between  these  two  groups 
are,  of  course,  the  best  candidates  for  a revised  predictor  battery. 

One  way  to  decrease  the  probability  of  faking  predictor  battery  responses 
is  to  administer  the  battery  to  Navy  or  Marine  Corps  personnel  well  before 
the  time  they  might  be  considered  for  recruiting  duty.  For  example,  upon 
becoming  a third-class  petty  officer,  individuals  in  the  Navy  might  be 
routinely  required  to  complete  the  predictor  battery.  Scores  on  the 
battery  could  be  stored  and  used  subsequently  for  selection  purposes 
if  the  man  or  woman  was  being  considered  for  a recruiter  assignment. 

Developing  Additional  Measures  of  Valid  Constructs 

Based  on  results  of  this  study  and  the  studies  reviewed  earlier  in 
this  report,  certain  constructs — e.g.,  traits,  personal  characteristics, 
and  interest  patterns — appear  to  be  associated  with  military  recruiter  ef- 
fectiveness. Additional  measures,  items  and  scales  should  be  developed  to 
tap  the  constructs  that  seem  to  be  valid  indicators  of  recruiter  perfor- 
mance. Extending  the  coverage  of  these  constructs  should  add  to  the  reli- 
ability of  measurement  and  to  the  validity  of  the  battery. 
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16PF  VALIDITY  STUDY 


Procedures  Related  to  Validating  the  16PF 

The  I6PF  Is  a personality  inventory  developed  by  Cattell  (Cattell,  Eber, 

&  Tatsuoka,  1970).  It  has  been  used  In  a variety  of  applied  settings  (e.g. , 
personnel  selection,  occupational  placement,  vocational  guidance,  clinical 
diagnoses,  assessment  of  accident  proneness,  and  school  performance  prediction) 
The  inventory  yields  scores  on  the  following  16  dichotomous  personality  traits: 


Low  Score 

Factor  Description 

1 Reserved,  detached,  critical, 
aloof,  stiff 

2 Dull,  low  Intelligence 

3 Affected  by  feelings,  emotion- 
ally less  stable,  easily  upset 
changeable 

4 Humble,  mild,  easily  led, 
docile,  accommodating 

5 Sober,  taciturn,  serious 

6 Expedient,  disregards  rules 

7 Shy,  timid,  threat-sensitive 

8 Tough-minded,  self-reliant, 
realistic 

9 Trusting,  accepting  conditions 

10  Practical,  "down-to-earth" 

11  Forthright,  unpretentious, 
genuine  but  socially  clumsy 

12  Self-assured,  placid,  secure, 
complacent,  serene 

13  Conservative,  respecting 
traditional  ideas 

14  Group  dependent,  a "joiner" 
and  sound  follower 

15  Undisciplined  self-conflict, 
lax,  follows  own  urges,  care- 
less of  social  ruleB 

16  Relaxed,  tranquil,  torpid, 
unfrustrated,  composed 


High  Score 
Description 

Outgoing,  warmhearted,  easy- 
going, participating 

Bright,  high  intelligence 

Emotionally  stable,  mature,  faces 
reality,  calm 

Assertive,  aggressive,  competitive, 
stubborn 

Hsppy-go-lucky , en thus iast ic 

Conscientious,  persistent,  moral- 
istic, staid 

Venturesome,  uninhibited,  socially 
bold 

Tender-minded,  sensitive,  clinging, 
overprotected 

Suspicious,  hard  to  fool 

Imaginative,  bohemian,  absent-minded 

Astute,  polished,  socially  aware 

Apprehensive,  self-reproaching, 
insecure,  worrying,  troubled 

Experimenting,  liberal,  free- 
thinking 

Self-sufficient,  resourceful, 
prefers  own  decisions 

Controlled,  exacting  will  power, 
socially  precise,  compulsive, 
following  self-image 

Tense,  frustrated,  driven, 
overwrought 
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Scores  on  the  16PF  scale  were  available  for  Navy  recruiters  who  had  taken 
this  inventory  either  before  being  selected  for  recruiting  duty  or  during 
their  training  experience  after  selection.  Also  available  were  composite 
scores  formed  by  applying  a prediction  equation  used  by  the  Navy  Recruiting 
Command  (Krug,  1972).  The  prediction  equation  employed  empirically  derived 
weights  for  four  of  the  16PF  scales  and  several  other  variables.  Thus,  as  a 
spinoff  of  this  study  16PF  scale  scores  and  the  composite  scores  were  cor- 
related with  performance  criteria  for  the  191  Navy  recruiters  who  had  completed 
the  16FF  and  supplemental  questions  previously. 

16PF  Validation  Results 


Correlations  between  16PF  scale  scores  and  the  five  performance  criteria 
appear  in  Table  A-l.  Results  show  that  six  of  the  16  scales  are  related  to 
Selling  Skills  with  correlations  of  .20  or  greater.  Only  two  scales  correlate 
.20  or  greater  with  Human  Relations  Skills  and  four  correlate  in  the  .20s  with 
Overall  Performance.  No  16PF  scales  correlate  as  highly  as  .20  with  Administra- 
tive Skills  or  Production. 

Relationships  between  the  composite  and  each  criterion  also  appear  in  the 
table.  The  composite  correlates  .21  with  Selling  Skills  and  .19  with  Production. 
Correlations  between  the  other  three  criteria  and  the  16PF  composite  are  lower. 
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Table  A-l 


Correlations  Between  16PF  Scale  Scores 
and  the  Five  Criteria  (N  - 191) 


16PF  Scales 

Selling 

Skills 

Human 

Relations 

Skills 

Adminis- 

trative 

Skills 

Overall 

Perform- 

ance 

Produc- 

tivity 

1 

.22 

.18 

.02 

.14 

.03 

2 

.08 

.14 

.17 

.15 

.08 

3 

.01 

.09 

.00 

.05 

-.01 

4 

.20 

.12 

.11 

.20 

-.04 

5 

.27 

.21 

.02 

.23 

-.13 

6 

-.08 

.04 

.12 

.03 

-.11 

7 

.21 

.18 

.12 

.23 

-.07 

8 

-.22 

.09 

.06 

-.14 

.00 

9 

.19 

.08 

-.03 

.14 

.13 

10 

.09 

.01 

.06 

.08 

.16 

11 

-.06 

-.05 

.13 

.05 

-.01 

12 

.02 

.04 

-.01 

-.02 

.02 

13 

.19 

.09 

-.08 

.11 

-.08 

14 

-.20 

-.20 

-.05 

-.24 

-.05 

15 

-.06 

.06 

.15 

.04 

.05 

16 

-.05 

-.12 

-.06 

-.11 

-.02 

Composite  Scorea 

.21 

.04 

.15 

.15 

.19 

*See  Krug  (1972)  for  a description  of  this  composite,  formed  by  empirically 
keying  the  16PF  and  several  other  variables. 
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SCALE  DEVELOPMENT  STRATEGIES  AND  DEFINITIONS  OF  RATING  DIMENSIONS 


SCALE  DEVELOPMENT  STRATEGIES  AND  DEFINITIONS  OF  RATING  DIMENSIONS 


1.  Trait  dimensions.  A literature  review  of  selection  studies  directed 
at  sales  jobs  and  interviews  with  several  Navy  recruiters  suggested  several 
traits  that  appeared  to  be  important  for  performing  effectively  as  a Navy 
recruiter.  An  attempt  was  made  to  summarize  the  trait  list  by  combining 
traits  or  eliminating  traits  when  two  or  more  of  them  were  close  in  meaning. 

These  procedures  resulted  in  eight  trait  dimensions:  Initiative,  Judgment, 
Imagination  and  Originality,  Personal  Impact,  Confidence,  Achievement  Orienta- 
tion, Organization,  and  Warmth. 

2.  Behavior  dimensions.  In  a previous  study  (Borman,  Hough,  & Dunnette, 
1976)  40  Navy  recruiters  and  21  of  their  immediate  supervisors  provided  over 
1000  behavior  examples  of  effective,  adequate,  and  ineffective  recruiter  per- 
formance. These  examples  were  content -analyzed  by  three  psychologists,  the 
process  yielding  nine  dimensions  of  recruiter  performance.  The  behavior 
examples  were  subsequently  retranslated  by  an  independent  sample  of  77  recruiters 
using  methods  outlined  by  Smith  and  Kendall  (1963) . Eight  behavlorally  anchored 
dimensions  survived  the  retranslation  process;  one  dimension,  salesmanship 
skills,  was  dropped  because  in  categorizing  examples  confusion  arose  between 
that  dimension  and  several  others.  The  final  seven  dimensions  were  Locating  and 
Contacting  Qualified  Prospects,  Gaining  and  Maintaining  Rapport,  Obtaining  In- 
formation from  Prospects  and  Making  Good  Person-Navy  Fits,  Establishing  and 
Maintaining  Good  Relationships  in  the  Community,  Providing  Knovledgable  and 
Accurate  Information  about  the  Navy,  Administrative  SkillB,  and  Supporting  Other 
Recruiters  and  the  Command. 

3.  Multidimensional  scaling  dimensions.  A similarity  judgment  task  was 
developed  to  discover  the  underlying  dimensionality  of  the  content  of  behavior 
examples  written  for  the  Navy  recruiter  job.  Sixty  behavior  examples  were 
selected  randomly  from  all  of  those  rated  as  reflecting  effective  performance, 
and  all  possible  pairs  of  these  examples  (1770  in  all)  were  generated  and 
randomly  assigned  without  replacement  to  20  different  protocols  so  that  each 
protocol  contained  approximately  89  different  item  pairs.  Thus,  each  of  the 
1770  possible  pairs  of  behavior  examples  appeared  once  in  one  of  these  20  pro- 
tocols. The  same  procedure  was  used  to  develop  four  more  sets  of  20  protocols, 
resulting  in  each  item  pair  appearing  exactly  five  times  in  100  protocols.  In 
addition,  for  each  protocol,  14  or  15  item  pairs  were  randomly  selected  (again 
without  replacement)  and  repeated  within  each  protocol  in  such  a way  that  they 
appeared  at  least  20  items  apart  from  the  same  item  pair.  These  repeat  item 
pairs  allowed  for  an  estimate  of  within-rater  reliability. 

Each  item  pair  in  the  protocols  required  the  respondent  to  assess  the 
similarity  of  the  job  facets  represented  by  the  two  behavior  examples.  A 
4-point  rating  scale  was  used  with  "4’'  Indicating  that  the  two  examples 
represented  completely  different  facets  of  the  recruiter  job  and  "1"  indicating 
that  the  two  examples  represented  identical  or  very  similar  facets  of  the  job. 

One  hundred  Navy  recruiters  completed  the  protocols,  with  16  of  them  per- 
forming the  task  unreliably  (£  < .50  for  the  correlation  between  the  14  or 
15  pairs  of  similarity  ratings).  Other  recruiters  responded  to  these  protocols 
until  every  one  was  reliably  completed. 


The  consistency  in  responses  across  recruiter  judges  was  then  assessed 
by  an  analysis  of  variance  (ANOVA)  comparing  the  variance  of  different  judges' 
similarity  ratings  of  the  same  item  pairs  with  the  total  variance  across  all 
similarity  ratings.  Such  an  ANOVA  procedure  was  possible  because  five  in- 
dependent estimates  of  the  similarity  of  each  of  1770  item  pairs  were  available. 
An  intraclass  correlation  coefficient  that  summarizes  this  across-judge  agree- 
ment was  .71  (£  < .001,  df  ■ 1770,  7080),  indicating  that  judges  agreed  closely 
in  their  similarity  ratings  of  item  pairs.  Therefore,  mean  similarity  ratings 
were  computed  for  each  of  the  1770  item  pairs  and  used  to  form  a 60  x 60 
similarity  matrix. 

The  similarity  matrix  was  then  submitted  to  a nonmetric  multidimensional 
scaling  analysis  (Shepard,  1962;  Kruskal,  1964),  with  each  solution  rotated 
to  the  varimax  criterion.  Stress  values  and  lnterpretability  of  dimensions 
were  used  to  determine  the  preferred  solution.  The  five-dimension  solution 
provided  eight  interpretable  job  dimensions  because  three  of  the  dimensions 
were  bipolar.  The  resultant  eight  scales  were  Prospecting,  Gathering  Informa- 
tion about  Applicants,  Planning  and  Organizing  Own  Recruiting  Schedule  and 
Practices,  Expending  Extra  Effort  to  Aid  Applicants  or  Recruits,  Preparing 
Recruits  and  their  Parents  for  Navy  Life,  Salesmanship,  Developing  Productive 
Relationships  in  the  Community,  and  Gaining  and  Maintaining  Friendly  Relation- 
ships with  Prospects. 

4.  Factor  analysis  dimensions.  The  eight  behavior  dimensions  were  used 
in  a previous  study  (Borman,  Hough,  & Dunnette,  1976)  to  rate  the  performance 
of  24  Navy  recruiters  stationed  in  the  Mlnneapolis-St . Paul  district.  Super- 
visory, peer,  and  self  ratings  were  gathered,  the  ratings  were  pooled  across 
the  three  sources,  and  the  pooled  ratings  on  the  eight  dimensions  were  inter- 
correlated.  A principal-components  factor  analysis  of  these  Intercorrelations 
followed  by  a varimax  rotation  of  the  three-factor  solution  revealed  three 
conceptually  meaningful  dimensions  of  recruiter  performance.  The  factors  and 
the  proportion  of  common  variance  accounted  for  by  each  appear  below. 

I.  Prospecting  and  Selling  (36%) 

II.  Planning,  Organizing,  and  Administrative  Skills  (37%) 

III.  Developing  Good  Navy-Community  Relations  and  Expending  Extra 
Effort  to  Aid  Prospects  and  Recruits  (27%) 
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RATING  DIMENSIONS  AND  DEFINITIONS  OF  PRETEST  RATING  SCALES 


1.  Inlt iat ive . Displaying  a willingness  to  seek  out  and  to  accept 
responsibility;  working  well  on  one's  own. 

2.  Judgment.  Ability  to  handle  day-to-day  problems  and  situations  in  a 
practical  and  efficient  way;  using  information  wisely  in  making  decisions. 

3.  Imagination  and  Originality.  Being  resourceful  and  creative  in  determin- 
ing productive  courses  of  action. 

4.  Personal  Impact.  Charisma;  creating  a positive  first  impression. 

5.  Confidence.  Projecting  self-assurance  and  decisiveness  In  dealing  with 
others . 

6.  Achievement  Orientation.  Enjoying  Involvement  in  the  process  of  achieving 
significant  goals;  capacity  to  stick  with  a task  and  to  expend  onsiderable 
physical  and  mental  energy  to  attain  an  objective. 

7.  Organization.  Planning  ahead  effectively;  being  well- u , 1 and 

efficient  in  performing  tasks  and  duties. 

8.  Warmth.  Relating  easily  and  naturally  to  others;  being  friendly,  out- 
going, and  warm. 

9.  Locating  and  Contacting  Qualified  Prospects.  Prospecting  effectively; 
contacting  large  numbers  of  persons  likely  to  join  the  Navy;  getting  prospects 
into  the  office. 

Skillfully  using  the  telephone  referrals,  recruits,  advertising  ideas, 
and  special  events  to  contact  young  persons  eligible  for  Navy  service  and  get 
their  attention. 

Knowing  where  and  when  to  prospect;  ability  to  persist  in  prospecting 
and  following  up  on  leads  even  under  considerable  adversity. 

10.  Gaining  and  Maintaining  Rapport.  Being  hospitable  to  prospects  in  the 
office. 


Gaining  the  trust  and  respect  of  prospects. 

Adjusting  to  prospects'  styles  and  acting  appropriately  with  different 
types  of  prospects. 

11.  Salesmanship  Skills.  Skillfully  persuading  prospects  to  join  the  Navy; 
using  Navy  benefits  and  opportunities  effectively  to  sell  the  Navy. 

Closing  skills;  adapting  selling  techniques  appropriately  to  different 
prospects. 

Effectively  overcoming  objections  to  joining  the  Navy. 
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12.  Obtaining  Information  from  Prospects  and  Making  Good  Person-Haw 
Fits . Exercising  listening  skills;  making  accurate  judgments  about  prospects' 
needs,  programs  desired,  and  the  like,  based  on  good  interviewing  skills. 

Effectively  obtaining  information  about  prospects  from  other  sources 
(e.g.,  high  school  principal)  to  assess  their  qualifications  and  needs. 

Assessing  accurately  prospects'  eligibility  for  various  programs. 

13.  Establishing  and  Maintaining  Good  Relationships  in  the  Conm»'nity. 
Contacting  and  working  effectively  with  high  school  counselors,  newspaper 
editors,  radio  and  TV  personnel,  and  others  capable  of  helping  recruiters  to 
enlist  prospects. 

Building  a good  reputation  for  the  Navy  by  developing  positive  relation- 
ships with  persons  in  the  community;  presenting  a good  Navy  image  in  the  com- 
munity. 


Establishing  and  maintaining  good  relationships  with  families  of  pro- 
spects . 

14.  Providing  Knowledgeable  and  Accurate  Information  about  the  Navy.  Dis- 
playing considerable  knowledge  about  Navy  programs,  schools,  and  educational 
opportunities;  answering  questions  about  the  Navy  in  a competent  manner. 

Providing  accurate  information  about  Navy  life;  skillfully  relaying 
information  about  boot  camp  so  that  the  prospective  recruit  is  informed  about 
irtiat  to  expect  but  not  discouraged  from  joining  the  Navy. 

Being  up  to  date  on  Becruitlng  Manual  changes  and  on  other  directions 
pertaining  to  program  or  school  changes. 

15.  Administrative  Skills.  Planning  ahead;  organizing  time  efficiently. 
Completing  paperwork  accurately  and  on  time;  keeping  track  of  appointments. 

Not  wasting  time. 

16.  Supporting  Other  Recruiters  and  the  Command.  Coordinating  activities 
with  other  recruiters  to  maximize  the  productivity  of  the  station  and  district. 

Using  own  skills  and  time  to  support  other  Navy  recruiters  when  ap- 
propriate; providing  constructive  feedback  to  other  Navy  recruiters  concerning 
their  skills  and  style;  providing  helpful  tips  to  new  recruiters. 

Pitching  in  to  support  orders  and  directives  from  higher  levels. 

17.  Prospecting.  Using  various  prospecting  techniques  to  get  qualified 
young  persons  interested  In  joining  the  Navy. 

Effectively  using  other  persons  to  provide  referrals  or  to  help 
prospect  for  potential  recruits. 


18.  Gathering  Information  about  Applicants.  Checking  up  on  an  applicant’s 
background . 

Asking  appropriate  persons  about  applicants  in  order  to  gather  informa- 
tion by  which  to  accept  or  reject  them. 

19.  Planning  and  Organizing  Own  Recruiting  Schedule  and  Practices.  Establish- 
ing a realistic  and  efficient  recruiting  time  schedule  to  meet  goals. 

Effectively  analyzing  own  recruiting  techniques. 

20.  Expending  Extra  Effort  to  Aid  Applicants  or  Recruits.  Willingly  offer- 
ing and  providing  assistance  to  recruits  or  applicants  who  have  a problem  or 
need. 

Helping  other  recruiters  to  meet  a recruit’s  or  applicant’s  special 

needs. 

21.  Preparing  Recruits  and  Their  Parents  for  Navy  Life.  Fully  informing 
recruits  and  their  parents  about  boot  camp  and  Navy  life. 

Developing  good  relationships  with  the  families  of  recruits. 

22.  Salesmanship.  Using  knowledge  of  Navy  jobs  and  the  qualifications 
necessary  for  those  jobs  to  sell  the  Navy. 

Listening  to  the  prospect  and  then  making  an  appropriate  and  effective 
sales  pitch. 

23.  Developing  Productive  Relationships  in  the  Community.  Volunteering  for 
community  events  or  projects  in  order  to  enhance  the  image  of  the  Navy  and 
Marine  Corps  image. 

Developing  relationships  with  recruits  and  with  persons  in  the  community 
as  sources  of  referrals. 

24.  Gaining  and  Maintaining  Friendly  Relationships  with  Prospects.  Being 
friendly,  outgoing,  and  sincere  with  all  prospects  regardless  of  their  back- 
grounds. 


Adapting  recruiting  style  to  the  individual  prospect. 

25.  Prospecting  Skills.  Effectively  contacting  qualified  prospects  and 
interesting  them  in  the  Navy. 

Locating  potential  recruits  through  other  persons  who  provide  referrals, 
free  advertising  and  the  like. 

Knowing  how,  when,  and  where  to  use  prospecting  resources  such  as  the 
telephone,  referrals,  recruits,  and  advertising  ideas  to  get  the  attention  of 
young  persons  qualified  for  Navy  service. 
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26.  Selling  Skills.  Skillfully  using  knowledge  of  Navy  benefits,  pro- 
grams, schools,  and  educational  opportunities  to  sell  the  Navy  to  prospects. 

Effectively  answering  prospects'  questions  about  the  Navy  and  over- 
coming their  objections  to  entering  the  naval  service. 

Using  selling  techniques  that  get  qualified  prospects  into  the  Navy. 

27.  Human  Relations  Skills.  Using  good  listening  and  interviewing  skills 
with  prospects. 

Effectively  adjusting  to  different  types  of  prospects  and  gaining  their 

trust. 


Providing  support  to  the  station,  zone,  district,  and  to  other  recruiters 
by  attending  more  to  organizational  quotas  than  personal  quotas  and  by  working 
to  enhance  team  effort  with  fellow  recruiters. 

28.  Likability. 1 Overall  subjective  judgment  of  how  well  you  like  this 
individual. 

29.  Overall  Effectiveness  as  a Recruiter.  Overall  level  of  effectiveness 
displayed  in  performing  as  a Navy  or  Marine  Corps  recruiter. 


1The  Likability  dimension  was  used  for  special  research  purposes  only  and 
was  not  Included  as  a performance  criterion  meesure  in  any  subsequent  analyses. 
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APPENDIX  C 

MAIN  SAMPLE  PERFORMANCE  RATING  SCALE  PACKAGE 

(For  Navy  only;  Che  Marine  Corps  package  was 
Che  same  excepC  for  reference  Co  Che  "Marine 
Corps"  racher  chan  Che  "Navy.") 
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GUIDELINES  FOR  MAKING  NAVY  RECRUITER  JOB  PERFORMANCE  RATINGS 
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navy  recruiter  rating  scale 

(EXAMPLE)1 


\ 


rating  scale  of  this  type  was  provided  for  each  of  the  performance 
dimensions  Jed  in  the  main  study.  The  summary  definition  at  the  top  of  the 
sheet  was  the  only  explanation  provided  for  the  supplemental  dimensions. 


4 % 


A.  LOCATING  AND  CONTACTING  QUALIFIED  PROSPECTS 


"Prospecting"  effectively;  contacting  large  numbers  of  persons  lively  to  join  the 
Navy;  skillfully  using  the  telephone,  referrals,  recruits,  advertising  ideas, 
special  events,  etc.,  to  contact  and  get  the  attention  of  young  persons  eligible 
for  Navy  service;  knowing  where  and  when  to  prospect;  ability  to  persist  in 
prospecting  and  following  up  on  leads  even  under  considerable  adversity;  getting 
prospects  into  the  office. 


Recrui ters 


r 
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Ineffective  Marginal 

Performance  Performance 

12  3*5 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ P 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ P 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ Q □ □ 

□ □ □ □ □ 

□ D □ □ O 
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APPENDIX  D 

NAVY  RECRUITER  BIOGRAPHICAL  AND  OPINION  SURVEY 
(Trial  Battery) 
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NAVY  RECRUITER  BIOGRAPHICAL  AND  OPINION  SURVEY 


Name 


1.  Your  age  is  . 

2.  How  many  years  have  you  served  in  the  Navy?  years 

3.  How  many  months  have  you  served  at  your  present  station?  months 

4.  How  many  months  have  you  served  within  the  district?  months 


SECTION  1:  In  this  section,  circle  the  one  alternative  that  describes  you  best. 


Sex 

a. 

Male 

b. 

Female 

Race 

a. 

White 

b. 

Black 

c. 

Spanish-American 

d. 

Oriental 

e. 

Native  American 

f. 

Other 

Height 

a. 

5*4"  or  shorter 

b. 

5’ 5"  to  5’7” 

c. 

5' 8"  to  5' 10” 

d. 

5’ 11"  to  6’1" 

e. 

6' 2"  to  6' 4" 

f. 

6’ 5"  or  taller 

Weight 

a. 

120  lbs.  or  less 

b. 

121  lbs.  to  140  lbs 

c. 

141  lbs.  to  160  lbs 

d. 

161  lbs.  to  180  lbs 

e. 

181  lbs.  to  200  lbs 

f. 

201  lbs.  to  220  lbs 

8- 

221  lbs.  to  240  lbs 

h. 

241  lbs.  or  more 

5.  How  far  did  you  go  in  school? 

a.  8th  grade  or  less 

b.  9 to  11  years 

c.  Graduated  from  high  school 

d.  1 to  4 years  of  college 

e.  Bachelor '8  degree 

f.  Other 
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6.  What  was  your  marital  status  at  the  tine  you  applied  to  be  a recruiter? 

a.  Single. 

b.  Separated. 

c.  Divorced. 

e.  Widowed. 

f.  Remarried. 

g.  Married. 

7.  What  was  your  father's  education? 

a.  Grade  school  or  less. 

b.  Some  high  school. 

c.  Completed  high  school. 

d.  Some  college. 

e.  Completed  college. 

f.  A graduate  degree  (M.A.,  M.S.,  Ph.D.,  etc.). 

8.  What  was  your  mother's  education? 

a.  Grade  school  or  less. 

b.  Some  high  school. 

c.  Completed  high  school. 

d.  Some  college. 

e.  Completed  college. 

f.  A graduate  degree  (M.A.,  M.S.,  Ph.D.,  etc.). 

9.  How  many  different  towns  did  you  live  in  before  entering  the  Navy? 

a.  1 

b.  2 

c.  3 

d.  4 

e.  5 

f.  6 

8.  7 

h.  8 or  more 

10.  In  what  section  of  town  did  your  family  live  longest  while  you  were 
growing  up? 

a.  An  exclusive  section  of  town. 

b.  In  a good  but  not  the  best  section. 

c.  Average  section  of  town. 

d.  In  one  of  the  poorer  sections  of  town. 

e.  In  a rural  area. 
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11.  How  many  times  did  you  change  schools  before  you  were  16  years  of  age? 
(Other  than  by  graduation) 

a.  Never. 

b.  Once. 

c.  Twice. 

d.  Three  times. 

e.  Four  or  more  times. 

12.  What  size  high  school  did  you  attend? 

a.  Fewer  than  100  students. 

b.  100-499  students. 

c.  500-999  students. 

d.  1000-2000  students. 

e.  More  than  2000  students. 

13.  As  a youngster,  how  often  were  you  a leader  in  your  group  of  friends? 

a.  Always. 

b.  Frequently. 

c.  Occasionally. 

d.  Seldom  or  never. 

e.  Was  not  a member  of  a group. 

f.  Can't  remember. 

14.  When  you  were  a child,  did  you  feel  that  you  received  adequate  recognition 
from  your  teachers  for  your  work  in  school? 

a.  Almost  always. 

b.  Usually,  but  not  always. 

c.  In  a moderate  amount. 

d.  Sometimes,  but  not  often. 

e.  Almost  never. 

15.  It  is  typical  for  you  to  live,  eat,  and  breathe  your  job  (in  jobs  you  held 
prior  to  recruiting  duty). 

a.  Definitely  agree. 

b.  Probably  agree. 

c.  Not  sure. 

d.  Probably  disagree. 

e.  Definitely  disagree. 

16.  When  you  were  a high  school  student,  were  you: 

a.  One  of  the  most  popular  students. 

b.  More  popular  than  most  students. 

c.  About  as  popular  as  most  students. 

d.  Not  quite  as  popular  as  most  students. 
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17.  How  often  have  you  done  extra  work  for  a job  which  was  not  really  required 
of  you? 

a.  Almost  every  day. 

b.  Several  times  a week. 

c.  About  once  a week. 

d.  Once  every  few  weeks. 

e.  About  once  a month  or  less. 

18.  In  group  discussion  or  meetings,  do  you  generally: 

a.  Take  an  active  role  in  discussions. 

b.  Remain  relatively  inactive,  but  try  to  learn  from  the  discussion. 

c.  Remain  relatively  Inactive,  offering  comments  only  rarely. 

d.  Rarely  or  never  attend  meetings. 

19.  About  the  best  Indication  of  a man's  worth  is  how  well  he  does  his  job. 


a.  Definitely  agree. 

b.  Probably  agree. 

c . Mot  sure . 

d.  Probably  disagree. 

e.  Definitely  disagree. 


20.  In  high  school,  when  friends  came  to  you  with  their  personal  problems, 
how  likely  were  you  to  go  out  of  your  way  to  give  them  help  or  advice? 

a.  Much  more  likely  than  most  people. 

b.  More  likely  than  most  people. 

c.  About  the  same  as  other  people. 

d.  Less  likely  than  most  people. 

e.  Much  less  likely  than  most  people. 

21.  On  the  average  how  much  sleep  do  you  require  to  feel  really  good? 

a.  Less  than  5 hours. 

b.  5 to  6-1/2  hours. 

c.  6-1/2  to  7 hours. 

d.  7 to  8 hours. 

e.  8 to  9 hours. 

f.  More  than  9 hours. 


22.  In  high  school  did  you: 

a.  Have  many  friends. 

b.  Have  a few  close  friends. 

c.  Have  hardly  any  friends. 

d.  Have  no  one  you  could  call  a friend. 

23.  Have  you  ever  done  door-to-door  (not  store)  selling? 

a.  Yes. 

b.  Mo. 
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24.  Do  you  make  a list  of  things  to  do  when  you  know  you  will  have  a busy 
day? 


a.  Yes,  always. 

b.  Yes,  usually. 

c.  Yes,  sometimes. 

d.  Yes,  but  only  rarely. 

e.  No,  never. 

25.  How  well  have  you  felt  you  were  able  to  understand  the  feelings  of  others? 

a.  Very  well. 

b.  Pretty  well. 

c.  Fairly  well.  J 

d.  Not  very  well. 

26.  How  often  do  you  find  that  your  first  impression  of  a person  is  the  right 
one? 


a.  Always. 

b.  Often. 

c.  Occasionally. 

d.  Rarely. 

e.  Never. 


27.  How  well  do  you  like  to  be  around  other  people? 

a.  1 enjoy  being  with  others  very  much;  only  rarely  do  I like 
to  be  by  myself. 

b.  I usually  enjoy  being  around  others,  occasionally  preferring 
to  be  by  myself. 

c.  I like  being  around  other  people  sometimes  and  at  other  times 
I like  to  be  by  myself. 

d.  I prefer  being  by  myself  and  only  occasionally  enjoy  being 
around  other  people. 


28.  I notice  little  things  about  a person  or  a situation  that  others  overlook. 


a.  This  happens  to  me  almost  all  the  time. 

b.  This  often  happens  to  me. 

c.  This  has  happened  to  me  several  times,  but  I wouldn't  say 
this  is  generally  true  of  me. 

d.  This  very  seldom  happens  to  me. 

e.  This  never  happens  to  me. 

29.  Some  people  easily  become  involved  in  a task  while  others  seldom  really 
"dig  into”  a task  or  job.  How  Involved  do  you  usually  become  in  a task 
or  job? 

a.  I often  have  trouble  sticking  with  It;  other  things  almost 
always  seem  to  come  up  to  distract  my  attention. 
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b.  I sometimes  become  involved  in  a task  or  job  that  interests 
me,  but  if  I'm  not  very  interested,  I seldom  become  involved. 

c.  I often  become  heavily  involved  in  a task  or  job. 

d.  I almost  always  become  engrossed  in  tasks  or  jobs. 

30.  Do  you  feel  you  are  a good  detail  person? 

a.  Definitely  yes;  I am  very  detail  oriented  and  attend  closely 
to  "nitty-gritties"  of  a task  or  job. 

b.  I am  probably  about  average  on  detail  orientation. 

c.  Not  really;  I tend  to  overlook  small  details  or  fall  to  do 

a really  thorough  job  of  attending  to  the  details  required  on 
many  tasks  or  jobs. 

d.  Definitely  no;  I often  miss  important  details  on  a task  or  job 
and  I'm  much  better  at  things  requiring  little  or  no  detail 
work. 

31.  How  organized  are  you  when  working  on  a job  or  task? 

a.  I plan  and  organize  my  work  almost  to  a fault. 

b.  I am  quite  well  organized  on  most  tasks  and  jobs. 

c.  Though  I generally  get  the  job  done,  I must  admit  my  organiza- 
tion could  usually  be  better. 

d.  Frankly,  I am  very  poor  at  planning  and  organizing,  and  prefer 
to  play  it  by  ear  when  working  on  a task  or  job. 

32.  Vhlle  you  were  growing  up,  which  of  the  following  types  of  persons  did  you 
especially  admire?  (Circle  all  that  apply.) 

a.  Religious  leaders. 

b.  Counselors  or  social  workers. 

c.  Coaches. 

d.  Military  leaders. 

e.  Political  leaders. 

f.  Parents. 

g.  Businessmen. 

h.  Doctors. 

1.  Lawyers. 

j.  Union  leaders. 

k.  Movie  or  TV  stars. 

l.  Teachers. 

m.  Tough  guys. 

n.  None  of  the  above. 

33.  When  you  were  in  high  school,  did  you  participate  in  any  of  the  following 
clubs,  societies,  or  activities?  (Circle  all  that  apply.) 

a.  Dramatics,  debating,  or  speech  clubs. 

b.  Fraternity  or  social  groups. 

c.  Music,  band,  chorus,  orchestra,  etc. 

d.  History  or  foreign  language  clubs. 

e.  Math  or  science  clubs. 
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f.  Literary,  magazine,  or  newspaper. 

g.  Team  sports  or  other  sports. 

h.  Student  government. 

i.  None  of  the  above. 

34.  In  which  of  the  following  groups  or  social  organizations  did  you  hold 
office  before  coming  on  recruiting  duty  (president,  secretary,  chairman 
of  the  committee,  etc.)?  (Circle  all  that  apply.) 

a.  Athletic  and  recreational  clubs — bowling,  golf,  tennis,  chess, 
bridge,  photography,  etc. 

b.  Fraternal  and  cultural  societies — Elks,  Masons,  K of  C,  IOOF, 

YMCA,  college  fraternity,  dramatics,  debating,  bible  class,  etc. 

c.  Civic  and  political  organizations — Rotary,  Kiwanls,  Lions, 

Chamber  of  Commerce,  Young  Republicans,  American  Legion,  PTA,  etc. 

d.  Student  government. 

e.  Never  held  any  offices. 

35.  Which  of  the  following  were  you  active  in  during  high  school?  (Circle  all 
that  apply.) 

a.  National  Honor  Society. 

b.  Future  Farmers  of  America. 

c.  Boy  Scouts  or  Girl  Scouts. 

d.  Church  youth  group. 

e.  Future  Teachers  of  America. 

f.  Cheerleaders  group. 

g.  Marching  band. 

h.  Football  team. 

1.  Basketball  team. 

j . Baseball  team. 

k.  Golf  team. 

l.  Tennis  team. 

m.  Hockey  team. 

n.  Wrestling  or  boxing. 

o.  Gymnastics  team. 

p.  Swimming  team. 

q.  Public  service  organizations  or  volunteer  work  (such  as  in 
hospitals). 

r.  None  of  the  above. 

36.  Listed  below  is  a series  of  24  statements  about  different  things  you  may  or 
may  not  have  done  and  feelings  you  may  have  had  about  various  situations 
and  people.  Please  read  each  one  carefully  and  decide  whether  or  not  the 
statement  is  true  or  mostly  true  about  you.  Circle  any  statement  that  is 
more  true  about  you  than  it  is  false. 

a.  You  graduated  from  high  school. 

b.  You  have  been  suspended  or  expelled  from  school. 

c.  You  dropped  out  of  high  school. 

d.  You  worked  instead  of  going  to  school. 

e.  As  a child,  you  hated  being  ordered  around  at  home  or  at  school. 

f.  You  ran  away  from  home  once. 

g.  As  a child,  you  hated  working  around  the  house. 
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h.  Your  parents  nay  have  tried,  but  they  did  a poor  job. 

i.  It  bothers  you  to  have  your  family  move  around  as  much  as 
military  families  do. 

j . You  worked  full  time  before  you  were  sixteen. 

k.  You  had  a social  security  card  when  you  were  sixteen. 

l.  Obeying  strict  rules  and  regulations  bothers  you. 

m.  It  bothers  you  to  be  taken  off  jobs  before  they  are  finished. 

n.  You  try  to  do  well  in  things  that  don't  interest  you. 

o.  You  were  never  fired  from  a job. 

p.  You  have  a bank  account. 

q.  From  twelve  to  fourteen,  you  usually  went  around  with  a group 
of  three  or  four  guys. 

r.  You  went  to  a beer  party  before  you  were  sixteen. 

s.  You  smoked  regularly  before  you  were  sixteen. 

t.  You  have  hitchhiked  more  than  200  miles. 

u.  You  have  been  in  a gang  fight. 

v.  You  have  been  arrested. 

w.  You  were  in  jail  overnight  once. 

x.  It  is  Important  to  you  to  be  stationed  near  a large  city. 
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REVISION  OF  THE  BIOGRAPHICAL  SURVEY 
(Example) 

29.  Some  people  easily  become  Involved  in  a task  while  others  seldom  really 
"dig  into"  a task  or  job.  How  Involved  do  you  usually  become  In  a task 
or  job? 

a.  I often  have  trouble  sticking  with  It;  other  things  almost 
always  seem  to  come  up  to  distract  my  attention. 

b.  I sometimes  become  involved  in  a task  or  job  that  interests 
me  greatly,  but  most  of  the  time  I quickly  lose  interest. 

c.  I often  become  heavily  Involved  in  a task  or  job  provided 
it  * 8 of  interest  to  me. 

d.  I almost  always  become  engrossed  in  tasks  or  jobs. 

Forty-three  Individuals  in  the  pretest  sample  responded  (c)  to  this  question, 
while  only  three  responded  (a),  five  responded  (b)  and  six  responded  (d).  Since 
the  item  was  clearly  not  differentiating  well  among  persons  in  the  sample,  two 
of  the  alternatives  were  changed  in  an  attempt  to  increase  variability  in  respond- 
ing to  the  item.  The  revised  item  appears  below. 

29.  Some  people  easily  become  Involved  in  a task  while  others  seldom  really 
"dig  into"  a task  or  job.  How  Involved  do  you  usually  become  in  a task 
or  job? 

a.  I often  have  trouble  sticking  with  it;  other  things  almost 
always  seem  to  come  up  to  distract  my  attention. 

b.  I sometimes  become  Involved  in  a task  or  job  that  Interests 
me,  but  if  I'm  not  very  Interested,  1 seldom  become  Involved. 

c.  I often  become  heavily  involved  in  a task  or  job. 

d.  I almost  always  become  engrossed  in  tasks  or  jobs. 
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DESCRIPTION  OF  THE  MONTE  CARLO  CROSS-VALIDATION  PROCEDURE 

The  Monte  Carlo  cross-validation  procedure  has  been  developed  over  a period 
of  several  years  by  Dr.  Rodney  L.  Rosse  of  the  research  staff  of  Personnel  i 

Decisions  Research  Institute.  Its  purpose  is  to  provide  cross-validity  estimates 
that  are  stable  in  comparison  to  other  sample-splitting  procedures  commonly  used  > \ 

to  index  the  cross-validity  of  sets  of  predictors  in  test  validation  research. 

This  appendix  first  reviews  briefly  the  critical  issues  related  to  nmlHng  I 

cross-validity  estimates,  discusses  the  approach  taken  by  the  Monte  Carlo  pro- 
cedures to  address  these  issues,  and  outlines  how  these  procedures  were  applied  i 

to  data  from  the  study  described  in  this  report.  j 

Most  methods  for  estimating  the  cross-validity  of  predictors  involve  "sample- 
splitting" in  one  form  or  another.  The  basic  procedure  used  in  these  "sample- 
splitting"  techniques  is  characterized  as  follows: 

1.  The  total  sample  is  divided  into  two  subsamples  at  random;  one  subsample 
may  be  called  a "developmental"  subsample  and  the  other  may  be  called  a "crosr- 
validation"  subsample. 

2.  The  set  of  statistical  rules  applied  to  the  total  sanq>le  to  obtain  the 
prediction  function  is  fully  repeated  on  the  "developmental"  subsample,  result- 
ing in  an  "interim"  prediction  function  that  is  only  to  be  used  for  cross-validity 
estimation. 

3.  This  "interim"  function  is  applied  to  the  computation  of  point  predictions 
for  each  subject  in  the  "cross-validation"  subsample. 

4.  The  appropriate  measure  of  association  (usually  a Pearsonian  coefficient) 
is  computed;  that  is  considered  the  estimate  of  validity  (i.e.,  cross-validity) 
for  the  prediction  function. 

Clearly,  the  validity  estimated  in  this  manner  is  conditional  upon  the  oc- 
currence of  a particular  split  and  thus  is  neither  a unique  nor  an  efficient 
estimator,  '.t  is  not  unique  because  simple  repetition  of  the  process  with  a 
new  random  split  will  result  in  a different  cross-validity  estimate.  Therefore, 
there  is  "within-sample"  variability  in  the  statistic.  It  is  not  efficient 
because  the  within-sample  variability  contributes  to  the  random  sampling  dis- 
tribution of  the  validity  estimate. 

The  Monte  Carlo  cross-validation  procedure  provides  a unique  and  efficient 
estimator  of  the  validity  of  an  empirically  determined  prediction  function  by 
using  the  computational  capability  of  a scientific  computer  to  do  the  above  four 
steps  repeatedly  with  rerandomization  of  the  "developmental"  and  "cross-validation" 
split  for  each  repetition,  and  reestimation  of  the  constants  for  the  prediction 
function.  The  process  gives  rise  to  a distribution  of  "cross-validities,”  the 
median  or  mean  of  which  nay  be  taken  as  the  estimates  of  validity  (referred  to 
here  as  the  Monte  Carlo  cross-validity  estimator). 

The  number  of  repetitions  is  determined  by  the  standard  error  of  the  mean 
of  the  "within-sample"  distribution.  According  to  the  central  limit  theorem, 
this  standard  error  decreases  as  a function  of  the  reciprocal  of  the  square  root 


of  the  number  of  0..  38-validity  estimates  obtained,  i.e.,  the  number  of 
repetitions.  Therefore,  the  researcher  may  simply  specify  the  maximum  within- 
s ample  error  he  wishes  to  have  in  the  average  of  the  estimators  of  validity, 
and  then  repeat  the  cross-validation  procedure  a sufficient  number  of  times 
to  reduce  the  standard  error  below  that  level.  Thus,  within  the  limits  of  this 
error,  the  final  validity  estimate  is  a unique  and  efficient  variance  estimator. 

Now  to  comment  briefly  on  an  approach  to  selecting  predictor  measures  in 
test  validation  research,  an  approach  that  influences  significantly  the  outcome 
of  Monte  Carlo  analyses.  It  has  proven  generally  useful  to  select  a relatively 
few  predictors  with  maximum  a priori  potential  for  predicting  performance.  In 
particular,  the  following  "rules**  have  been  used  in  conducting  validation 
research  with  the  Monte  Carlo  method: 

1.  The  variables  to  be  introduced  into  the  estimation  process  are  kept  to 
a minimum  and  restricted  to  those  for  which  there  is  some  a priori  reason 

to  believe  that  they  would  be  useful  (even  if  just  a "hunch**) . 

2.  The  manner  in  which  the  variable  or  item  is  to  be  used  in  the  prediction 
function  is  predetermined  as  much  as  possible  (e.g.,  a variable  may  be  constrained 
to  weight  only  in  a positive  or  negative  direction  or  a variable  may  be  assumed 

to  have  value  only  if  included  with  a "curvilinear"  fit) . 

3.  The  weighting  that  is  used  may  be  specified  arbitrarily  by  the  researcher 
instead  of  depending  upon  the  empirical  estimation  of  weights  (e.g.,  arbitrary 
unit  or  differential  weighting  of  items  or  scales). 

4.  Items  that  are  to  be  empirically  keyed  may  be  rescaled  and  several 
response  categories  lumped  together  when  indicated  by  judgment  on  an  a priori 
basis. 

5.  The  use  of  multiple  regression  and,  particularly,  step-wise  multiple 
regression,  is  avoided  where  data  are  insufficient  to  obtain  highly  stable 
estimates  of  the  covariances  among  predictor  variates.  This  is  because  these 
regression  methods  are  greatly  influenced  by  the  covariance  estimates  and  would 
not  be  expected  to  replicate  where  Inadequate  data  are  used  in  the  estimation. 

In  the  analysis  of  data  for  the  present  project,  there  was  a particularly 
good  basis  for  a priori  keying  of  items  and  scales  because  the  information 
obtained  in  the  pilot  phase  was  available  to  aid  the  researchers  in  making  these 
keying  decisions.  Therefore,  two  keying  strategies,  one  for  each  of  the  two 
different  types  of  predictor  data,  were  used  to  make  a priori  predictor-criterion 
matches.  The  types  of  data  were  (1)  biographical  history  items,  and  (2)  interest 
inventory  items. 

The  biographical  history  items  were  empirically  keyed  (for  each  criterion) 
according  to  the  following  rules:  ' 

1.  A response  (or  group  of  responses)  for  a particular  item  was  weighted 
4-1,  -1,  or  0 according  to  whether  the  response  was  hypothesized  to  be  positively 
or  negatively  related  (or  unrelated)  to  performance  on  the  criterion. 
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2.  To  be  Included  in  a Monte  Carlo  run,  the  raan  of  the  criterion  scores 
for  those  endorsing  a response  was  required  to  fall  outside  of  the  48  to  52 
range  in  T-score  values  (T-score  means  ■ 50,  standard  deviations  “ 10)  and 
the  mean  criterion  score  was  required  to  be  in  the  proper  direction. 

3.  And  finally,  in  order  for  a response  to  be  considered  for  inclusion 
in  a Monte  Carlo  run,  the  proportion  of  respondents  who  endorsed  it  was  re- 
quired to  be  greater  than  or  equal  to  .05  of  the  half  sample. 

For  each  Monte  Carlo  run,  then,  item  responses  were  selected  on  the  basis 
of  these  rules  and  the  resultant  key  was  cross-validated  on  the  other  half  of 
the  sample.  The  cross-validity  coefficient  for  each  run  was  formed  by  computing 
a single  predictor  score  for  each  Individual  in  the  cross-validation  sample 
using  the  key  developed  on  that  run  and  correlating  those  scores  with  the  per- 
formance scores  obtained  by  the  same  individuals.  The  median  of  these  cross- 
validity coefficients  then  formed  our  final  validity  estimate  for  a key  that 
was  developed  using  total  sample  data. 

In  the  procedure  used  for  estimating  cross-validity  for  the  Interest  in- 
ventory, each  item  was  treated  as  an  ordered  scale  with  three  points,  cor- 
responding to  the  like-indifferent-dislike  format  of  the  Strong  Campbell  In- 
terest Inventory  used  in  the  study.  A priori  judgments  about  items  were  made 
according  to  pretest  data  analysis  results  and  conceptual  considerations.  For 
these  a priori  selections  of  items  for  each  criterion,  it  was  specified  that 
each  item  should  be  considered  positively  related,  negatively  related,  to  have 
no  predetermined  direction  but  to  be  included  in  the  keys,  or  not  to  be  in- 
cluded in  the  keys.  Thus,  for  each  criterion  a pool  of  items  was  developed 
that  appeared  to  show  promise  for  predicting  performance  on  that  criterion. 

These  a priori  judgments  were  based  upon  an  analysis  of  pretest  data  and  con- 
ceptual considerations.  Further,  most  but  not  all  of  the  items  selected  in 
this  manner  were  given  directional  constraints. 

For  each  Monte  Carlo  run,  a Fearsonian  correlation  coefficient  between  each 
item  in  the  pool  and  the  criterion  was  computed  using  half  the  sample.  Items 
were  Included  in  the  key  by  the  following  steps: 

1.  The  direction  of  the  correlation  for  an  item  was  required  to  be  con- 
sistent with  the  a priori  constraints. 

2.  The  items  were  rank  ordered  according  to  the  absolute  values  of  cor- 
relations with  the  criterion,  and  the  top  50  percent  were  selected  for  the  key. 

As  with  the  biographical  items,  for  each  Monte  Carlo  run,  prediction  scores 
were  computed  for  members  of  the  cross-validation  sample  using  the  key  developed 
for  that  run  and  these  scores  were  correlated  with  the  criterion  performance 
scores.  Again,  the  median  of  these  cross-validity  coefficients  served  as  the 
final  validity  estimate  for  a key  formed  on  the  basis  of  total  sample  data. 
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HYPOTHESIZED  PREDICTOR-PERFORMANCE  CRITERION  RELATIONSHIPS 

The  following  tables  present  hypothesized  relationships  between  the  in- 
ventory scales  and  items  selected  for  predictor  composites  and  the  pretest 
performance  criteria.  In  all  of  the  tables,  the  intended  direction  of  the  item 
is  provided:  a minus  sign  (-)  indicates  a negative  direction;  a plus  sign 
(+)»  a positive  direction;  and  the  numeral  one  (1),  a keyed  item  with  no 
direction  given.  For  the  Monte  Carlo  program,  the  (+)  and  (-)  keyed  items 
were  constrained  to  be  keyed  in  the  Intended  direction,  while  (1)  items 
could  be  keyed  in  either  direction  in  the  Monte  Carlo  runs.  See  Appendix 
E for  a description  of  the  Monte  Carlo  method.  In  the  Marine  Corps  sample, 
the  keys  presented  here  were  cross-validated  directly.  Thus,  that  key 
does  not  include  the  items  keyed  (1). 
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Table  F-l  (Continued) 


Table  F-2 


Hypothesized  SCII  Item-Performance  Criterion  Relationships 


Item 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 
29 


Prospecting  Planning  & Good  Relations 

& Selling  Organizing  with  Community 


+ 

+ + + 

+ + + 

+ + 

+ + 


+ 

+ 


+ 

+ 


+ 


1 1 
1 + 1 

+ + 

1 
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Table  F-2  (Continued) 


Item 


Prospecting  Planning  & Good  Relations 

& Selling  Organizing  with  Comnunity 


30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 


+ 

1 

1 


+ 

+ 

+ 


1 

+ 

1 


+ 


1 

+ 

+ 


1 

1 

1 

1 


+ 

+ 

+ 

1 

1 

+ 

1 

+ 

1 


+ 

1 


1 

1 

1 

1 

+ 


1 


1 


+ 


1 

1 
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Table  F-2  (Continued) 


Table  F-2  (Continued) 


Prospecting 

Item  & Selling 


Planning  & Good  Relations 

Organizing  with  Community 


94 

95 

96 

97 

98 

99 
100 
101 
102 

103 

104 

105 

106 

107 

108 

109 

110 
111 
112 

113 

114 

115 

116 

117 

118 

119 

120 
121 
122 

123 

124 


+ 


1 

1 


1 

1 


1 

1 

+ 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


+ 

1 


A 


r-8 


Table  F-2  (Continued) 


Item 

125 

126 

127 

128 

129 

130 

131 

132 

133 

134 

135 

136 

137 

138 

139 

140 

141 

142 

143 

144 

145 

146 

147 

148 

149 

150 

151 

152 

153 

154 

155 


Prospecting  Planning  & Good  Relations 

& Selling  Organizing  with  Community 


+ 


+ 


+ 


+ 


+ + 

1 

1 


1 


1 + 
1 
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Table  F-2  (Continued) 


I 

I 

i 

i 

i 


Item 

157 

158 

159 

160 
161 
162 

163 

164 

165 

166 

167 

168 

169 

170 

171 

172 

173 

174 

175 

176 

177 

178 

179 

180 
181 
182 

183 

184 

185 

186 

187 

188 


Prospecting 
& Selling 


1 


1 


+ 


♦ 


Planning  & 
Organizing 


+ 


+ 


■f 


♦ 


Good  Relations 
with  Community 


+ 


WO 


• 1 


i 

( 


i. 
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Table  P-2  (Continued) 


I 

i 

i 


Prospecting  Planning  & 

Item  & Selling  Organizing 


221 

222 

+ 

223 

224 

225 

226 

227 

- 

228 

229 

230 

231 

232 

233 

234 

1 

235 

236 

237 

238 

239 

1 

240 

241 

242 

1 

243 

244 

1 

245 

246 

+ 

247 

248 

249 

- 

250 

251 

+ 

252 

♦ 

+ 


+ 


+ 


♦ 

+ 

+ 


Good  Relations 
with  Community 


A 


* 
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Table  F-2  (Continued) 


Item 

253 

254 

255 

256 

257 

258 

259 

260 
261 
262 

263 

264 

265 

266 

267 

268 

269 

270 

271 

272 

273 

274 

275 

276 

277 

278 

279 

280 
281 
282 

283 

284 


Prospecting 
& Selling 


+ 


1 

1 


1 


Planning  & Good  Relations 

Organizing  with  Community 


+ 


+ 
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Tabl«  F-2  (Continued) 


Item 

285 

286 

287 

288 

289 

290 

291 

292 

293 

294 

295 

296 

297 

298 

299 

300 

301 

302 

303 

304 

305 

306 

307 

308 

309 

310 

311 

312 

313 

314 

315 


Prospecting  Planning  & Good  Relations 

& Selling  Organizing  with  Community 


+ 

+ + 


+ 


+ 


+ 

1 

+ + 


1 


+ 


1 
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Table  F-2  (Continued) 


Item 

Prospecting 
& Selling 

Planning  & 
Organizing 

Good  Relations 
with  Community 

316 

+ 

317 

+ 

318 

- 

319 

- 

320 

- 

- 

321 

322 

323 

1 

1 

324 

- 

- 

- 

325 
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Table  P-3 


Hypothesized  Biographical  Survey  Item-Performance  Criterion  Relationships 


Item 


Prospecting  & Selling 

fit  Two  Overall  Perform-  Good  Relations  Planning  & 

ance  Criteria  with  Community  Organizing 


1 

2 

3 

4 

Section  1 
1 
2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 


1 

1 

1 


3 


b 


1.  2 


5 


5 


5 


1,  2 

5 


*(+)  means  the  response  was  keyed  +,  (-)  it  was  keyed  -. 
^Refers  to  item  response  alternative. 
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2 


1 

3 

3,  4 


1 

2 

1 


22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 


1 


1 


2 

1 

17 


3,  4 1 3,  4 


3 

4,  5 


2 

1 4,  5 1 4,  5 


4,  5 
2 

5 1 

17 


4,  5 
2 

5 15 

17 


a(+)  means  the  response  was  keyed  +,  (-)  it  was  keyed 
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APPENDIX  G 


IMPLEMENTATION  OF  THE  PREDICTOR  TEST  BATTERY 
FOR  USE  IN  SELECTING  NAVY  AND  MARINE  CORPS  RECRUITERS 


IMPLfMENTATION  OF  THE  PREDICTOR  TEST  BATTERY 
FOR  USE  IN  SELECTING  NAVY  AH)  MARINE  CORPS  RECRUITERS 

Final  composites  of  personality  scales  have  been  developed  for  each 
criterion.  Separate  composites  were  developed  for  the  Navy  and  Marine  Corps 
using  data  from  the  total  samples  of  Navy  and  Marine  Corps  recruiters.  That 
is,  the  best  and  most  complete  information  available  about  relationships 
between  the  personality  scales  in  the  battery  and  performance  on  the  various 
criteria  was  used  to  select  final  sets  of  scales  for  each  criterion  for  the 
Navy  and  another  series  of  final  sets  of  scales  for  each  criterion  for  the 
Marine  Corps.  This  selection  of  personality  scales  stands  the  greatest  chance 
of  providing  valid  predictor  composites  for  each  of  the  performance  criteria 
because  it  makes  use  of  the  maximum  amount  of  empirical  information  available 
about  personality  scale-performance  relationships. 

Likewise,  the  SCII  and  Biographical  and  Opinion  Survey  items  that  appeared 
most  promising  for  predicting  recruiter  effectiveness  on  each  performance 
criterion  were  selected  according  to  relationships  between  item  responses 
and  performance  on  each  criterion.  Separate  selections  were  made  for  the 
Navy  and  Marine  Corps  using  data  from  the  total  samples.  Thus,  the  per- 
sonality composites  and  keys  for  the  SCII  and  Biographical  and  Opinions 
Survey  are  ready  to  go.  These  composites  and  keys  are  not  the  same  ones 
as  those  listed  in  Appendix  F,  the  hypothesized  predictor-criterion  rela- 
tionships. They  have  been  drawn  from  that  list  and  contain  only  the  items 
and  scales  that  the  study  found  to  be  empirically  related  to  recruiter 
performance. 1 This  means  that  the  predictor  battery  may  be  shortened  from 
its  present  length  for  operational  use,  since  several  of  the  items  and  scales 
did  not  prove  to  be  valid  and  therefore  may  be  eliminated  from  the  battery. 

The  reduction  in  length  should  cut  testing  time  to  approximately  1-1/2  hours. 

How  can  the  Navy  or  Marine  Corps  use  the  predictor  battery  in  practice? 

The  battery  should  be  used  both  for  diagnostic  purposes  and  for  helping  to  make 
final  selection  decisions  on  candidates  for  the  recruiter  job.  First,  scoring 
the  battery  should  prove  straightforward.  Templates  that  fit  over  answer 
sheets  can  easily  be  constructed,  allowing  for  quick  scoring  of  the  battery. 

Two  clerks,  working  independently  of  one  another,  should  score  the  responses 
for  each  candidate  and  check  each  other's  scores  to  ensure  the  accuracy  of 
final  scores.  Further,  the  raw  scores  should  be  transformed  to  Navy  standard 
scores  (T-s cores  with  a mean  of  50  and  standard  deviation  of  10)  to  make  the 
predicted  performance  levels  more  interpretable  for  Navy  and  Marine  Corps 
decision  makers.  The  clerks  could  also  make  this  transformation  using  a simple 
conversion  table. 

Alternatively,  the  battery  could  be  machine-scored  at  a central  location. 
This  approach  would  require  that  respondents  use  machine-scorable  answer  sheets 
when  completing  the  battery. 

Either  scoring  approach  is  capable  of  yielding  standardised  predicted  per- 
formance scores  for  candidates  on  each  of  the  three  performance  dimensions— 

1The  final  composites  and  keys  are  not  displayed  in  this  rsport  for  security 
reasons.  They  have  been  delivered  to  NAVPKRSRAKDCEN , Navy  Recruiting  Command, 
mad  officials  at  Marine  Corps  Headquarters. 
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Selling,  Human  Halations,  and  Administrative  Skills.  For  the  Marine  Corps, 
such  scores  can  also  be  computed  for  the  criteria  Overall  Performance  and 
Production,  while  a score  on  the  criterion  Overall  Performance  (but  not 
Production)  can  be  estimated  for  the  Navy.  These  predicted  performance  scores 
can  be  plotted  on  a display  sheet  for  each  candidate  in  a manner  similar  to 
that  shown  in  the  example  in  Figure  G-l.  This  information  can  then  be  used 
both  diagnostically  and  as  an  aid  in  making  a final  select  or  reject  decision.2 

To  point  up  its  potential  use  as  a diagnostic  tool,  suppose  that  a Navy 
recruiting  district  had  a couple  of  locations  in  which  the  Navy  was  not  well 
accepted,  and  it  was  deemed  important  to  get  into  those  locations  recruiters 
who  possessed  good  human  relations  skills.  Zf  the  district  saw  fit  to  place 
new  recruiters  into  those  locations,  district  decision  makers  might  be  well 
advised  to  attend  to  candidates'  predicted  performance  in  the  human  relations 
area.  The  candidate  whose  scores  are  depicted  in  Figure  6-1,  for  example, 
would  not  be  a very  good  choice  for  such  a placement.  Of  course,  the  general 
level  of  the  profile  and  predicted  effectiveness  in  the  Overall  Performance 
criterion  should  obviously  be  weighted  heavily  in  selecting  Navy  or  Marine  Corps 
recruiters. 


2 The  Marina  Corpe  should  not  includs  predicted  performance  scores  for  both 
Overall  Performance  and  Production  on  a display  sheet  because  the  information 
may  be  confusing.  Since  these  two  criteria  are  both  suasmry  measures  of  ef- 
fectiveness, the  predicted  scores  for  the  two  criteria  should  be  averaged  and 
displayed  as  a single  score. 
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A.  J.  Candidate 


Predicted  Performance 

Selling  Human  Relations  Administrative 

Skills  Skills  Skills 


Overall 

Performance 


80 

70 

60 

50 

40 

30 

20 


* 


* 
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Figure  G-l 


Example  of  a possible  design  for  a display  system  to  depict  a 
candidate's  predicted  performance  as  a recruiter. 
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